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Interpreting the evidence from meta-analysis for the impact of 
digital technology on learning 


Overview 


The aim of this review is to present a synthesis of the evidence from meta-analysis about the 
impact of the use of digital technology in schools on children’s attainment, or more widely the 
impact of digital technology on academic achievement. It is divided up into three main 
sections. The first sets out an overview of the wider research into the impact of technology on 
learning to set the context and the rationale for the value of this information. The next section 
reviews the evidence from meta-analysis and other quantitative syntheses of research into 
the impact of digital technology. A further section looks at trends in the use of digital 
technology and learning in the UK and internationally, to provide further context for the 
recommendations which follow. 


The purpose of this review is to identify implications for future investment in the use of digital 
technology for learning in schools. Digital technologies are now embedded in our society. 
Focus has shifted from whether or not to use them in teaching and learning, to understanding 
which technologies can be used for what specific educational purposes and then to 
investigate how best they can be used and embedded across the range of educational 
contexts in schools. 


Summary of key points 


Overall, the research evidence over the last forty years about the impact of digital 
technologies on learning consistently identifies positive benefits. The increasing variety of 
digital technologies and the diversity of contexts and settings in which the research has been 
conducted, combined with the challenges in synthesising evidence from different 
methodologies, makes it difficult to identify clear and specific implications for educational 
practice in schools. 


Studies linking the provision and use of technology with attainment tend to find consistent but 
small positive associations with educational outcomes. However a causal link cannot be 
inferred from this kind of research. It seems probable that more effective schools and 
teachers are more likely to use digital technologies more effectively than other schools. We 
need to know more about where and how it is used to greatest effect, then investigate to see 
if this information can be used to help improve learning in other contexts. We do not know if it 
is the use of technology that is making the difference. 


Research findings from experimental and quasi-experimental designs — which have been 
combined in meta-analyses — indicate that technology-based interventions tend to produce 
just slightly lower levels of improvement when compared with other researched interventions 
and approaches (such as peer tutoring or those which provide effect feedback to learners). 
The range of impact identified in these studies suggests that it is not whether technology is 
used (or not) which makes the difference, but how well the technology is used to support 
teaching and learning. There is no doubt that technology engages and motivates young 
people. However this benefit is only an advantage for learning if the activity is effectively 
aligned with what is to be learned. It is therefore the pedagogy of the application of 
technology in the classroom which is important: the how rather than the what. This is the 
crucial lesson emerging from the research. 


Taken together, the correlational and experimental evidence does not offer a convincing case 
for the general impact of digital technology on learning outcomes. This is not to say that it is 
not worth investing in using technology to improve learning. But it should encourage us to be 
cautious in the face of technological solutions to educational challenges. Careful thought is 
needed to use technology to best effect. 


There is a recurrent and specific challenge in understanding and applying research evidence 
as it takes time for robust evidence to emerge in education, and the rapid pace of change of 
technology makes this difficult to achieve. 


The challenge is to ensure that technology is used to enable, or make more efficient, effective 
teaching and learning practices. With this in mind the findings from the synthesis of the meta- 
analyses indicate the following overall trends: 


Collaborative use of technology (in pairs or small groups) is usually more effective than 
individual use, though some pupils, especially younger children, may need guidance in 
how to collaborate effectively and responsibly. 


Technology can be as powerful as a short but focused intervention to improve learning, 
particularly when there is regular and frequent use (about three times a week) over the 
course of about a term (5 - 10 weeks). Sustained use over a longer period is usually less 
effective at improving this kind of boost to attainment. 


Remedial and tutorial use of technology can be particularly practical for lower attaining 
pupils, those with special educational needs or those from disadvantaged backgrounds in 
providing intensive support to enable them to catch up with their peers. 


In researched interventions, technology is best used as a supplement to normal teaching 
rather than as a replacement for it. This suggests some caution in the way in which 
technology is adopted or embedded in schools. 


Tested gains in attainment tend to be greater in mathematics and science (compared with 
literacy for example) though this is also a more general finding in meta-analysis and may 
be at least partly an artefact of the measurement process. In literacy the impact tends to 
be greater in writing interventions compared with reading or spelling. 


At least a full day’s training or on-going professional inquiry-based approaches to support 
the introduction of new technology appear the most successful. The implication is that 
such support should go beyond the teaching of skills in technology and focus on the 
successful pedagogical use of technology to support teaching and learning aims. 


Overall, the over-arching implication is that the technology is solely a catalyst for change. The 
question is how can technology can bring about improvement and make teaching and 
learning practices more efficient or effective. Focusing on the change (and the process of 
change), in terms of learning is essential in supporting effective use. 


Recommendations 


The rationale for the impact of digital technology on teaching and learning needs to be 
clear: 
e Will learners work more efficiently, more effectively, more intensively? Will the 
technology help them to learn for longer, in more depth, more productively? Or 
will the teacher be able to support learners more efficiently or more effectively? 


The role of technology in learning should be identified: 
e Will it help learners gain access to learning content, to teachers or to peers? Will 
the technology itself provide feedback or will it support more effective feedback 
from others, or better self-management by learners themselves? 


Technology should support collaboration and effective interaction for learning: 
e The use of computer and digital technologies is usually more productive when it 
supports collaboration and interaction, particularly collaborative use by learners 
or when teachers use it to support discussion, interaction and feedback. 


Teachers and/or learners should be supported in developing their use of digital 
technology to ensure it improves learning. 


e Training for teachers (and for learners), when it is offered, usually focuses on 
technology skills in using the equipment. This is not usually sufficient to support 
teachers and pupils in getting the best from technology in terms of their learning. 
On-going professional development and support to evaluate the impact on 
learning is likely to be required. 


5. Identify what learners and teachers will stop doing: 
e The use of digital technology is usually more successful as a supplement rather 
than as a replacement for usual teaching. Technology is not introduced into a 
vacuum. It is therefore important to identify carefully what it will replace or how 
the technology activities will be additional to what learners would normally 
experience. 


Approach and methods 


This review summarises the research evidence contained in meta-analyses to identify 
patterns of impact in the accumulating research about the effects of technology on learning, 
and to identify the extent of the possible impact of technology on learning. A systematic 
search revealed 48 studies which synthesised primary research studies of the impact of 
technology on the attainment of school age learners (5-18 year olds). Whilst this presents 
only a partial and retrospective view of such impact, it is the only approach to allow a 
systematic comparison of a large number of studies with an estimate of the extent of the 
effects on learning. 


Background 


The role of technology in education has been an important question since the potential of 
computer technology to transform Skinner’s teaching machines was recognised in the 1960s. 
It remains an important issue today with debates about the impact of technology on our 
society, the implications of quick and easy online access to information for knowledge and 
learning and the effect of technology on young people’s social, emotional and physical 
development frequently in the news. It is therefore important to take stock of what we know 
about the impact of digital technology on education from what we have learned over the last 
fifty years. Appendix 1 sets out a number of these issues in terms of some contemporary 
myths about the effects of technology. 


The main approach used to evaluate the impact of technology on teaching and learning in 
schools has been where pupils’ attainment across a range of tested curriculum outcomes has 
been correlated with the quantity or quality of technology which was available or which they 
experienced in their institutions (see, for example, Watson, 1993; Wenglinsky, 1998; Weaver, 
2000; BECTA 2003). In the USA, only a small relationship between computer use in the 
school curriculum and improvement in pupils’ test scores was found in a longitudinal study 
(Weaver, 2000). At this very general level, computer use makes very little difference to pupils’ 
achievement. In the UK, the Impact 2 study (Harrison et al. 2004) identified statistically 
significant findings positively associating higher levels of ICT use with school achievement at 
each Key Stage, and in English, Maths, Science, Modern Foreign Languages and Design 
Technology. An association between high ICT use and higher pupil attainment in primary 
schools was also reported in an earlier Teacher Training Agency study (Moseley et al. 1999, 
p 82) though the interpretation by the research team was that more effective teachers (and 
more effective schools) tended to use more innovative approaches, or chose to use the ICT 
resources that they had more appropriately, rather than that the technology itself was the 
cause of the differences in pupil performance. 


This connection between technology and learning is found fairly consistently however, and 
other studies have indicated a stronger association. The ICT Test Bed evaluation identified a 
link between high levels of ICT use and improved school performance. The rate of 
improvement was faster in ICT Test Bed Local Authorities (LAs) than in equivalent 
comparator LAs in KS2 English (Somekh et al. 2007). However, what this association shows 
is that, on average, schools with higher than average levels of ICT provision also have pupils 
who perform slightly higher than average. The causal link could be quite the reverse, with 
high performing schools more likely to be better equipped or more prepared to invest in 
technology or more motivated to bring about improvement. Fuchs and Woessmann’s (2004) 
analysis of this link between provision and performance based on the Programme for 
International Student Assessment (PISA) data supports this interpretation: 


“the initial positive pattern on computer availability at school simply reflects that 
schools with better computer availability also feature other positive school 
characteristics. Once these are controlled for, computer availability at school is 
not related to pupil performance in math and reading.” (p. 13) 


The Organisation for Economic Co-operation and Development's (OECD) more detailed 
analysis of Programme for International Student Assessment (PISA) data indicates a complex 
picture of association between pupil performance, their access to computers at home and at 
school together with frequency of use which varies from country to country (OECD 2006, p 
51-66). Though as a note of caution the research found that pupils who used computers most 
widely tended to perform slightly worse on average than those with moderate usage. Overall 
the analysis suggests that the linkage may not be a simple causal one, nor necessarily a 
simple linear association. There may be a limit to the amount of technology which is 
beneficial. 


In findings from experimental and quasi-experimental research studies, where gains in 


knowledge or understanding for groups of pupils using ICT has been compared with gains for 
groups learning the same content without technology, results again tend to show positive 
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benefits for ICT. These have been reviewed using a narrative approach with consistently 
positive conclusions (e.g. Parr & Fung, 2000; Andrews et a/. 2002; Cox et al. 2004; Hartley, 
2007) as well as through quantitative synthesis using meta-analysis (see Appendix 2 (2000- 
2012) and 3 (1990-1999) for more details about these studies, with a full bibliographical list of 
meta-analyses of the impact of digital technologies on learning in Appendix 4). Again these 
reviews typically conclude that technology has a positive and measurable effect on learning. 
Most of these reviews of the efficacy of ICT or digital technologies do not, however, consider 
the effects comparatively. A large majority of researched educational interventions have a 
positive impact but the relative impact is not usually considered (see, for example, Hattie, 
Biggs & Purdie, 1996; Sipe & Curlette, 1997; Marzano, 1998; Hattie, 2008). When a 
comparative view is taken technology interventions appear to be less beneficial, as Sipe and 
Curlette (1997) originally observed: 


“when compared to ‘no computers’, ‘computers’ produces a nice effect size. 
However, when compared with typical effect of innovation on educational 
achievement, computer innovations are not that different from the average 
innovation.” (p 608) 


Taken together, the correlational and experimental evidence does not offer a convincing case 
for the general impact of digital technologies on learning outcomes. Serious questions can be 
raised about the nature of the evidence base (Hrastinski & Keller, 2007). It may be the case, 
of course, that ICT and digital technologies do have an impact on learning, but that this is not 
apparent when looking at attainment (as measured by performance tests), or that it is 
particularly beneficial for certain groups or learners. It is therefore important to identify more 
precisely and articulate more clearly where and the use of digital technologies is beneficial 
(Schacter & Fagano, 1999). As the OECD study concludes: 


“More micro-studies are needed within countries to explore the extent to 
which for individual pupils, certain kinds of computer usage raise 
performance, and which kinds are most effective. At the same time, in 
countries where basic computer access is approaching universal, policy 
needs to turn its attention from providing the technology to ensuring that 
its usage is effective.” 

(OECD, 2006, p 69) 


The proliferation of technologies also makes this question hard to answer at a general level. 
One of the criticisms of the meta-analytic studies listed in Appendix 4, is that they tend to put 
all of the different kinds of technologies into a single category of ‘technology’ or ‘ICT’ begging 
the question of what the range of impact is, and whether some technologies or some 
educational approaches using technology are more effective than others. Similarly with 
correlational studies, it may be that some schools are using (Some) technologies to beneficial 
effect, but that when the data is aggregated, this is impossible to identify. 


A further, more speculative point relates to the phases of implementation or adoption of digital 
technologies. The evidence for this is more tentative and is based on a personal interpretation 
of trends over time. There appears to be a pattern of impact of ICT or digital technologies 
where in the early stages there is a high level of enthusiasm, supported by either anecdotal or 
qualitative accounts of the benefits of the introduction of a new or emerging technology in an 
educational setting, such as with integrated learning systems or interactive whiteboards. At 
the next stage, as the technology and teaching approaches develop and evolve, these effects 
are investigated more rigorously. At this stage a more mixed message tends to appear with 
different studies finding different effects or levels of effect (see for example, Parr and Fung’s 
(2000) retrospective analysis of Integrated Learning Systems or Higgins, Beauchamp and 
Miller’s (2007) review of interactive whiteboards). It is rare for further studies to be conducted 
once a technology has become fully embedded in educational settings as interest tends to 
focus on the new and emerging, so the question of overall impact remains elusive. 


If this is the situation, there may, of course, be different explanations. We know, for example, 
that it is difficult to scale-up innovation without a dilution of effect with expansion (Cronbach et 
al. 1980; Raudenbush, 2003; 2008). It may also be that early adopters (Rogers, 2003; Chan 


et al. 2006) tend to be tackling particular pedagogical issues in the early stages, but for later 
adopters (Rogers’s ‘early’ and ‘late majority’) then the focus may shift to the adoption of the 
particular technology itself, without it being chosen as a solution to a specific teaching and 
learning issue. At this point the technology may be the same, but the pedagogical aims and 
intentions are different, and this may explain a reduction in efficacy. 


—_  &4° eS 


INNOVATION ADOPTION LIFECYCLE 


Figure 1: Rogers’s adoption of innovation lifecycle’ 


Where this makes a further difference may also be in what the technology replaces. 
Technology is not introduced into a vacuum. As schools and teachers introduce technology 
they stop doing something else. When teachers choose to adopt technology themselves they 
often do it as part of a process of inquiry (Somekh, 2007) and it replaces or displaces some 
problematic practice; when it is adopted for its own sake, its displaces or replaces other 
teaching and learning activities which may have been as (or more) effective. Hence an 
ecological view of adoption is needed, where the justification of technology adoption is a 
relative one (Zhao & Frank, 2003). It should replace less effective practices, and be 
effectively integrated into the resources available to a learner to support their learning 
(Luckin, 2008), as part of a more effective or more efficient learning context. As yet we do not 
have the tools to enable us to support these decisions (Underwood and Dillon, 2004). 


Overall, the challenge of assessing the impact is more acute than ever. The rise in 
technologies and the range of ways that they can be used in diverse educational settings 
across the spectrum of learners, coupled with the pace of change of technology make the 
task ever more demanding. The focus must shift from the technologies to the pedagogies of 
use, and the analysis of general impact to the specific differences that digital technologies 
make to teaching and learning contexts and interactions with regard to different learners. The 
quantity of technology use is not the key factor to student learning. “How much” matters only 
when “what and how” are identified (Lei & Zhao, 2007). 


Global trends: a move towards increasing scepticism? 


The UK is pioneering in terms of the use of ICT and digital technologies in many areas of 
education, and in the schools sector in particular. Important contributions to the literature 
include reviews on: effective pedagogy in primary schools (Moseley et al. 1999), evidence 
about the impact of Integrated Learning Systems (see Parr & Fung, 2000), the effects of 
interactive whiteboards, (e.g. Higgins, Beauchamp & Miller, 2007), the use of mobile and 
handheld technologies (Cheung & Hew, 2006) and virtual learning environments (Passey & 
Higgins, 2011). It is therefore not surprising that current international research reiterates the 
broad messages outlined above, such as about the small overall association between 
technology and attainment (Wainer ef a/. 2008) and positive findings for smaller and more 
intensive interventions (Liao & Hao, 2008). 


' Image from http://en.wikipedia.org/wiki/File:DiffusionOflnnovation.png 
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One feature of the international research which is not reflected in the studies discussed above 
is the role of ICT and digital technologies in assessment. A considerable proportion of the 
published research in the field looks at computer-based testing (including the assessment of 
higher order thinking and assessment of extended writing) and computer-adaptive testing. 
Pedagogy, curriculum and assessment are inextricably linked (Mabry & Snow, 2006). The 
current situation in the UK perhaps indicates that this is an area for further research and 
development. The challenge will be to link work on pupils’ involvement in formative 
assessment, with effective diagnostic feedback for teachers, as well as the summative 
purposes and accountability issues (Harlen, 2007) involved in schools. 


There are some global trends identifiable which reflect enthusiasm for new and emerging 
technologies accompanied by more varied evidence as these technologies are adopted more 
widely (for an overview of evidence relating to schools see Voogt & Knezek, 2008): 


e Continuing enthusiasm for new and emerging technologies is unlikely to diminish as 
innovative technologies offer new teaching and learning opportunities (Web 2.0, 
mobile and ubiquitous technologies, multi-touch surfaces, learning analytics, cloud 
computing: e.g. Chan et a/. 2006) 

e —_ Identifying the impact of one-to-one provision of technology is challenging. This is 
both for laptops (Dunleavy et a/. 2007; Silvernail & Gritter, 2007) and mobile 
technologies (Naismith et al. 2004). Similarly, there is a challenge for one-to-one 
provision in terms of pedagogy, such as developing effective interaction and 
collaboration (Liu & Kao, 2007) or in addressing teachers’ concerns effectively 
(Donovan et al. 2007): for a review see Penuel (2006). This may be particularly 
pertinent to the current enthusiastic introduction of tablet computers and iPads. 

e The internet has had a relatively disappointing impact as an educational resource 
(e.g. Cole & Hilliard, 2006), especially considering concerns about its use (e.g. 
Richards et al. 2008). The ‘world-wide web’ is an amazing resource which has 
developed in the space of just over twenty years. The facility to search and find 
information in different forms about almost any subject matter you can think of is a 
fantastic educational resource, which would have been literally incredible 30 years 
ago. However letting learners loose on the internet is a little like sending teenagers 
into the British Library and expecting them to make successful forays to support their 
learning. 

e There is a lack of evidence of the beneficial impact of e-learning on pupils’ 
achievement. Much of the research published relates particularly to the Higher 
Education sector (e.g. Davis & Graf, 2005; Kanuka & Keland, 2008; Passey and 
Higgins, 2011), with very little evidence of impact on students’ learning. It is certainly 
the case that well motivated and experienced learners can learn very effectively 
through e-learning. It is also clear, however, that without such motivation, skills and 
experience e-learning may well not be so successful. 

e Enthusiasm for gaming and games-based approaches may be misplaced, as there is 
a lack of evidence of impact in terms of attainment (Vogel et al. 2006). Children and 
young people are often highly motivated by computer games and simulations. The 
challenge is to ensure that the learning can be applied outside of the game 
environment. 

e There are some concerns about the detrimental impact on health and well-being of 
sustained used of computer technology, particularly for younger learners (e.g. Straker 
et al. 2005). These concerns relate to physical issues (such as posture and eyesight); 
health concerns (such as physical fitness and obesity) and social issues (social 
isolation or addiction). We can’t “uninvent” new technologies, but we can think about 
using them in ways which promote physical and mental well-being. 

e There is an increasing acknowledgement of the tension between technological and 
pedagogical change (e.g. Steffens, 2008), and the influence of other aspects of the 
educational system (such as assessment in particular, e.g. Mabry & Snow, 2006). 
The pace of technological change in society has been very rapid over the last 50 
years or so, and appears to show no sign of slowing down. Aspects of schooling, 
such as teaching and learning, the curriculum and, perhaps most importantly, 
assessment and accountability have changed rather more slowly. The curriculum and 


its assessment in turn shape the way technology is (and can be) used in schools, 
arguably limiting the potential of new and emerging technologies for learning. 


One interpretation of the trends in the wider literature is a recognition of the seriousness of 
the challenge from enthusiasts (e.g. Underwood, 2004) to a growing critical voice from the 
skeptics (e.g. Oppenheimer, 2003; Wainer et al. 2008; Slay, 2008) with an increased interest 
in the cost-effectiveness or value for money of technology in education (e.g. Margolisa et al. 
2007) and the issue of sustainability (Mee, 2007), which can only be exacerbated in times of 
economic difficulty. This is a battle new and emerging technologies are likely to find hard to 
win, as early iterations of technologies tend to be more expensive than mass-produced 
models. If our speculation is correct that innovators and early adopters tend to get the best 
from such technologies, this sharpens the challenge. The majority who jump on the 
bandwagon of the technology (and get it cheaper), don’t necessarily know what to do with the 
equipment it to get the best from it educationally. If Rogers’ (2003) theory is correct, effect will 
diminish over time as the ‘late majority’ may also be more reluctant converts. On the other 
hand, it is impossible to imagine that digital technologies will not be used in educational 
settings as they are now so embedded in wider society. At this point the question of cost- 
effectiveness and relative benefit becomes increasingly urgent. Will schools be able to 
sustain the investment in interactive whiteboards, one-to-one provision of laptops, PDAs or 
iPads or the next generation of multi-touch desks and sustain the legacy equipment they 
already have? Do we have sufficient evidence to argue which older technologies should be 
retained and which might be replaced with more effective or more efficient approaches for 
teaching and learning with newer technologies? These challenges frame the context in which 
we currently find ourselves. 


Why meta-analysis? 

This review intentionally summarises the evidence contained in meta-analyses and other 
quantitative syntheses of research to identify patterns of impact in the accumulating research 
about the effects of technology on learning so as to draw possible implications for the future. 
Meta-analysis also allows an estimate to be made of the extent of the possible impact of 
technology on learning in terms of the effect sizes calculated. This helps to put the impact of 
technology in perspective, both in terms of its relative benefit, but also to identify how much 
more effective teaching and learning might be when supported with digital technologies. A 
systematic search of education databases and journals revealed 48 studies which 
synthesised primary research studies of the impact of technology on the attainment of school 
age learners (5-18 year olds). Whilst we accept this presents only a partial and retrospective 
view of such impact, we suggest it is the only review approach to allow a systematic 
comparison of a large number of studies together with an estimate of the extent of the effects 
on learning. 


Meta-analysis is a method of combining the findings of similar studies to provide an overall 
quantitative synthesis or ‘pooled estimate of effect’. The results of separate interventions 
using technology can be combined so as to identify clearer conclusions about which 
interventions are effective and which factors are associated with more effective approaches. 
The advantages of meta-analysis over other approaches to reviewing are that it combines or 
‘pools’ estimates from a range of studies and can therefore aggregate results to identify 
patterns or trends in findings over time. 


It can also show whether the findings from similar studies vary more that would be predicted 
from their samples so that the causes of this variation can be investigated (“moderator 
analysis”). This is particularly valuable as the results from a range of smaller studies can be 
combined to provide answers to questions without relying on the statistical significance of 
each of the individual studies as this relates closely to sample size. Many small studies with 
moderate or low effects may not reach statistical significance and if you review the field by 
simply counting how may were statistically significant, you may be misled into thinking that 
the evidence is less conclusive than if you combine these studies into one study or meta- 
analysis. The statistical techniques to undertake meta-analysis form a set of transparent and 
replicable rules which are open to scrutiny. 
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Another key advantage of meta-analysis is that it helps to deal with the quantity of information 
in research which can overwhelm other approaches. This is particularly important when trying 
to draw relative inferences across different areas of education research. The number of 
studies available to review in any area of technology and education is extensive, so 
techniques to aggregate and build up knowledge to propose further research and test 
theories and ideas are invaluable. 


We have identified that 45 meta-analyses of the effects of technology on learning in schools 
have been published between 1990 and 2012 (see Appendix 2 and Appendix 3). The most 
recent of these, 30 published since 1999 are summarised in Table 1 below’. We separated 
the analysis (1990-1999 and 2000-2012) to check that the findings and implications from 
earlier and possibly obsolete technologies were not influencing the overall findings. Meta- 
analysis is a retrospective approach, and the earliest meta-analyses in the 1980s reviewed 
the computer technology used in education from the 1960s to the 1980s. The kinds of 
technology and software have changed beyond recognition, though some of the approaches 
(such as ‘drill and practice’) are still recognisable. Overall it is questionable what can be 
inferred about digital technology use for current practice from the earliest experiments. One 
noticeable finding is that the typical overall effect size in the general analyses of the impact of 
technology on learning is that it is between 0.3 and 0.4, just slightly below the overall average 
for researched interventions in education (Sipe & Curlette, 1997; Hattie, 2008). However the 
range of effects is also very wide (-0.03 to +1.05) suggesting that it is essential that the 
differences between technologies and how they are used should be taken into account. 
Interestingly, there is no real change in this difference over time, suggesting that when 
technology is used to improve current practice, similar gains are achieved. 


Meta-analysis also lets us identify patterns in these findings to investigate whether larger 
effects are found with some kinds of technology, different approaches to using technology or 
their impact on different learners and in different contexts, rather than just identifying whether 
technology has a positive effect on average (Tamim et al., 2011). Looking for patterns or 
themes in this way may help identify where the use of new and emerging technologies are 
likely to be beneficial in the future. It may help us identify ‘best bets’ for learning (Higgins, 
Koktsaki & Coe, 2012). 


2 Summary tables of meta-analyses published between 2000 and 2012 and between 1990 and 1999 can also be 
found in the appendices. 
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Table 1: Summary of meta-analyses published between 2000 and 2012 


Focus Reference Overall ES | Impact on Notes 
General Camnalbur & Erdogan, 1.05 Academic success In Turkey 
2010 
General Christmann & Badgett, 0.34 Academic outcomes Elementary students 
2003 
General Liao, 2005 0.55 Student achievement In Taiwan 
General Tamim, 2009 0.35 Student achievement/ Second order meta-analysis (same data as Tamim et al. 2011) 
performance 
General Sandy-Hanson, 2006 0.24 School achievement Kindergarten to Grade 12 
General Waxman et al., 2002 0.39 Cognitive outcomes 
General Waxman et al., 2003 0.44 Cognitive outcomes Update/extension of Waxman et al. 2002 
Mathematics | Cheung & Slavin, 2011 0.15 On mathematics Kindergarten to Grade 12 
Mathematics | Li & Ma, 2010 0.71 On mathematics School students’ mathematics learning 
Mathematics | Seo & Bryant, 2009 NSPE® Mathematics performance Students with learning disabilities 
0.33 
(median) 
Mathematics | Tokpah, 2008 0.38 On mathematics Computer Algebra Systems 
Maths and Kulik, 2003 0.38 ILS on maths Instructional Technology in Elementary and Secondary Schools 
Science (median) Computer tutorials in science 
0.59 Simulations in science 
(median) Live ‘labs’ 
0.32 
(median) 
0.01 
(median) 
Science Bayraktar, 2001 0.27 Science achievement 
Science LeJeune, 2002 0.34 Lower order outcomes Simulated experiments in science education 
0.38 Higher order outcomes 
0.19 Retention follow up test 
Science Onuoha, 2007 0.26 Science academic Computer-Based Laboratory on College And Pre-College 
achievements Science 
Literacy Blok et al., 2002 0.25 Reading skills and/or Beginning reading 


>No single pooled effect 


comprehension 


Literacy Goldberg et al., 2003 0.50 Writing quantity Computers on student writing 
0.41 Writing quality 
Literacy Graham & Perrin, 2007 | 0.55 Writing quality Word processing on writing 
Literacy Kulik, 2003 0.30 Writing quality Elementary and secondary Schools 
0.43 Accelerated Reader 
0.06 Reading/ILS 
Literacy Moran et al., 2008 0.49 Reading Middle school grades 
Literacy Morphy & Graham, NSPE Word-processing programs and weaker writers/ readers 
2012 0.52 Writing quality 
0.48 Writing length 
Literacy Pearson et al. 0.49 Reading Middle school grades: same study as Moran et al., 2008 
Literacy Sisson, 2008 0.35 Academic performance Fast ForWord 
(mean) Standardised reading tests 
0.22 
(mean) 
Literacy Soe et al., 2000 0.26 Reading achievement 
Literacy Strong et al., 2011 NSPE Fast ForWord 
0.08 Vs untreated controls 
-0.03 Vs treated controls 
Literacy Torgerson & Elbourne, 0.37 Spelling 
2002 
Literacy Torgerson & Zhu, 2003 | NSPE ICT on literacy learning in English, 5-16 
0.89 Word processing on writing 
0.20 ICT on spelling 
0.28 Computer texts on reading 
Other focus | Cassil, 2005 0.43 Academic achievement Mobile and hand held technologies 
Other focus | Lou etal., 2001 0.16 Individual achievement Small group vs individual learning with tech 
Other focus | Means etal., 2009 0.24 Learning outcomes Online learning 
7 studies looked at K-12 students ES 0.16 
Other focus | Rosen & Salomon, 2007 | 0.11 Mathematics achievement Constructivist Technology-Intensive Learning Environments 
0.46 Constructivist vs traditional 
Other focus | Sitzman et al. 2006 0.15 Web-based instruction 
Other focus | Vogel et al., 2006 0.07 Cognitive gains Games and simulations 
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Identifying themes in the findings from meta-analysis 


A number of areas can be identified in the meta-analyses which have been systematically 
explored through moderator analysis. In some cases, the evidence is inconsistent or 
inconclusive so these themes should be considered as indicative and worthy of further 
research or exploration in schools. Frustratingly, it is not possible to identify if particular kinds 
of technologies or certain kinds of application (such as tutorial, practice software, use as a 
tool: Liao, 1992; Khalili & Shashaani 1994; Fletcher-Flinn & Gravatt 1995) are more effective. 
This variation suggests that how (or how well technology is used is the important 
consideration rather than the choice of a particular technology or a particular approach. 


Collaborative use of technology (in pairs or small groups) is usually more effective than 
individual use. This can be identified in separate meta-analyses (e.g. Liao, 2005) where it has 
been identified as a cause of variation, and as a general trend in technology studies (Lou et 
al. 2001). However some pupils, especially younger children, may need support in 
collaborating effectively. 


Technology can be used very effectively as a short but focused intervention to improve 
learning (Bayraktar 2001; Moran et al. 2008), particularly when there is regular and frequent 
use (about three times a week: Bangert-Drowns, 1993; Cheung & Slavin; 2011) over the 
course of about a term (5 -10 weeks: LeJeune, 2002; Sandy-Hanson, 2006). Sustained use 
over a much longer period is usually less effective at improving attainment (e.g. Liao 

1992; Sandy-Hanson, 2006). However the inconsistency in the evidence about duration and 
intensity of use makes it difficult to draw firm conclusions. 


Remedial and tutorial use of technology can be particularly effective for lower attaining pupils 
(Lou et al. 2001), or those with special educational needs (Li & Ma, 2010; Sandy-Hanson 
2006; Sisson, 2008) or those from disadvantaged backgrounds (Kuchler, 1998, but see also 
Cheung & Slavin, 2011) in providing intensive support to enable them to catch up with their 
peers. 


In researched interventions, technology is best used as a supplement to normal teaching 
rather than as a replacement for it (Liao, 1998; Khalili & Shashaani, 1994; Bayraktar, 2001 
Cheung & Slavin; 2011) Kulik, 2003; Sisson, 2008). This suggests some caution is the way in 
which technology is adopted or embedded in schools. 


Tested gains in attainment tend to be found across the curriculum with comparatively greater 
effects in mathematics and science (Khalili & Shashaani 1994; Fletcher-Flinn & Gravatt 1995 
Li & Ma, 2010; Seo & Bryant, 2009; Tokpah, 2008; Kulik, 2003; Bayraktar, 2001; LeJeune, 
2002; Onuoha, 2007). However, this is also a more general finding in meta-analysis (Hattie, 
2008) and may be at least partly a measurement artefact. In literacy, the impact tends to be 
greater in writing interventions (Goldberg et al., 2003; Kulik, 2003; Morphy & Graham, 2012) 
compared with reading (Blok et al., 2002; Soe et al., 2000) or spelling (Torgerson & Zhu, 
2003). 


Training and professional development for teachers is an important component of successful 
approaches. At least a full day’s support (Ryan, 1991) or on-going professional inquiry-based 
approaches appear the most successful (Conlon, 2004). The implication is that such support 
should go beyond teaching skills in technology use and focus on the effective pedagogical 
use of the technology to support teaching and learning aims (Cheung & Slavin, 2011). 


There is not a consistent picture about age with some meta-analyses finding inconsistent 
variation associated with age or school type (Liao, 1998; Roblyer, 1989; Goldberg et al. 2003; 
Li & Ma, 2010), and others not (Bayraktar 2001; Fletcher-Flinn & Gravatt 1995; Blok et al. 
2002: Cheung & Slavin; 2011). 


Of course, this is evidence about what has happened in the past, with older technologies and 
across a wide range of diverse settings and contexts. It is a record of what has worked, and is 
not a prediction of what the impact of new and emerging technologies will be. 


Conclusions and recommendations 


Overall, the research evidence over the last 40 years about the impact of computer and digital 
technologies on learning consistently identifies positive benefits. The increasing variety of 
digital technologies and the diversity of contexts and settings in which the research has been 
conducted, combined with the challenges in synthesising evidence from different 
methodologies make it difficult to identify clear and specific implications for educational 
practice in schools. 


Studies linking provision and use of technology with attainment tend to find consistent but 
small positive associations with educational outcomes. However, a causal link cannot be 
inferred from this kind of research. It seems probable that more effective schools and 
teachers are more likely to use ICT and digital technologies more effectively than other 
schools. We need to know more about where and how it is used to greatest effect, then 
investigate if this information can be used help to improve learning in other contexts. 


Research findings from experimental and quasi-experimental designs which have been 
combined in meta-analyses indicate that overall technology-based interventions tend to 
produce just slightly lower levels of improvement when compared with other researched 
interventions. The range of impact identified in these studies suggests that it is not whether 
technology is used (or not) which makes the difference, but how well the technology is used 
to support teaching and learning. This alignment of technology and learning is important. 
There is no doubt that technology engages and motivates young people. However this benefit 
is only an advantage for learning if the activity is effectively aligned with what is to be learned. 
It is therefore the pedagogy of use of technology which is important: the how rather than the 
what. 


With computer and digital technologies there is a recurrent and specific challenge in 
understanding and applying the research evidence as it takes time for robust evidence to 
emerge in education and the rapid pace of change of technology makes this difficult to 
achieve. With this in mind the findings from the synthesis of the 45 meta-analyses published 
since 1990 indicates the following overall trends: 


e Collaborative use of technology (in pairs or small groups) is usually more effective than 
individual use, though some pupils, especially younger children, may need support in 
collaborating effectively. 


e Technology can be used very effectively as a short but focused intervention to improve 
learning, particularly when there is regular and frequent use (about three times a week) 
over the course of about a term (5 -10 weeks). Sustained use over a longer period is 
usually less effective at improving attainment. 


e Remedial and tutorial use of technology can be particularly effective for lower attaining 
pupils or those with special educational needs or those from disadvantaged backgrounds 
in providing intensive support to enable them to catch up with their peers. 


e In researched interventions, technology is best used as a supplement to normal teaching 
rather than as a replacement for it. This suggests some caution in the way in which 
technology is adopted or embedded in schools. 


e Tested gains in attainment tend to be greater in mathematics and science (compared with 
literacy for example) though this is also a more general finding in meta-analysis and may 
be at least partly a measurement artefact. In literacy, the impact tends to be greater in 
writing interventions compared with reading or spelling. 
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e Training and professional development for teachers is an important component of 
successful approaches. At least a full day’s support or on-going professional inquiry- 
based approaches appear the most successful. The implication is that such support 
should go beyond teaching skills in technology use and focus on the effective 
pedagogical use of the technology to support teaching and learning aims. 


Overall the key implication is that the technology is solely a catalyst for change. What is it that 
teachers or learners actually do which brings about any improvement in learning? Focusing 
on the change (and the process of change) in terms of learning is essential in supporting 
effective use. 


Recommendations 


1. The rationale for the impact of digital technologies on teaching and learning needs to be 
clear: 
e Will learners work more efficiently, more effectively, more intensively? Will the 
technology help them to learn for longer, more deeply, more productively? Or will 
the teacher be able to support learners more efficiently or more effectively? 


2. The role of technology in learning should be identified: 
e Will it help learners gain access to learning content, to teachers or to peers? Will 
the technology itself provide feedback or will it support more effective feedback 
from others? 


3. Technology should support collaboration and effective interaction for learning: 
e The use of computer and digital technologies is usually more productive when it 
supports collaboration and interaction, particularly collaborative use by learners 
or when teachers use it to support discussion, interaction and feedback. 


4. Teachers and/or learners should be supported in developing their use of digital and 
computer technologies to ensure it improves learning. 
e = Skills training is not usually sufficient to support teachers and pupils in getting the 
best from technology. On-going professional development and support to 
evaluate the impact on learning is likely to be required. 


5. Identify what learners and teachers will stop doing: 

e The use of computer and digital technologies is usually more successful as a 
supplement rather than as a replacement for usual teaching. It is therefore 
important to identify carefully what it will replace or how the technology activities 
will be additional to what learners would normally experience. 


16 


References 


Andrews R., Burn A., Leach J., Locke T., Low G.D. & Torgerson C. (2002) A Systematic 
Review of the Impact of Networked ICT on 5-16 year olds’ literacy in English. |n: 
Research Evidence in Education Library. Issue 1. London: EPPICentre, Social Science 
Research Unit, Institute of Education. 

Becta (2003), Primary Schools — ICT Standards. An Analysis of National Data from Ofsted 
and QCA by Becta, Becta, Coventry. 
http://partners.becta.org.uk/index.php?section=rh&catcode=_re_rp_02_a&rid=13619 
(accessed 31/10/08). 

Chan, T., Roschelle, J., Hsi, S., Kinshuk, Sharples, M., Brown, T., Patton, C., Cherniavsky, J., 
Pea, R., Norris, C., Soloway, E., Balacheff, N., Scardamalia, M., Dillenbourg, P., Looi, 
C., Milrad, M., & Hoppe, U. (2006). One-to-one technology-enhanced learning: An 
opportunity for global research collaboration. Research and Practice in Technology 
Enhanced Learning, 1.1: 3-29. 

Cheung, W. S., & Hew, K. F. (2009). A review of research methodologies used in studies on 
mobile handheld devices in K-12 and higher education settings. Australasian Journal of 
Educational Technology, 25.2: 153-183. 

Cole, J.M. & Hilliard, V. R. (2006) The Effects Of Web-Based Reading Curriculum On 
Children’s Reading Performance And Motivation Journal of Educational Computing 
Research, 34.4: 353-380. 

Cox, M., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2003). A review 
of the research literature relating to ICT and attainment. Coventry: British Educational 
Communications and Technology Agency. 

Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R.O., Hornik, R. C., Phillips, D. C., 
Walker, D. F., & Weiner, S. S. (1980) Toward reform of program evaluation: Aims, 
methods, and institutional arrangements. San Francisco: Jossey-Bass. 

Davies, J. and Graff, M. (2005 Performance in e-learning: online participation and student 
grades British Journal of Educational Technology 36. 4: 657 — 663. 

Donovan, L., Hartley, K. & Strudler, N. (2007) Teacher Concerns During Initial Implementation 
of a One-to-One Laptop Initiative Journal of Research on Technology in Education 39. 
3: 263 — 269. 

Doyle, L.H. (2003) Synthesis through meta-ethnography: paradoxes, enhancements, and 
possibilities Qualitative Research 3; 321. 

Dunleavy, M. Dexter, S. & Heinecke W.F. What added value does a 1:1 student to laptop 
ratio bring to technology-supported teaching and learning? Journal of Computer 
Assisted Learning 23: 440-452. 

Fuchs, T. & Woessmann, L. (2004) Computers and student learning: bivariate and 
multivariate evidence on the availability and use of computers at home and at school 
CESIFO Working Paper No. 1321 Category 4: Labour Markets (Nov. 2004). Retrieved 
from: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=619101 

Harlen, W. (2007) Criteria For Evaluating Systems For Student Assessment Studies in 
Educational Evaluation 33: 15-28. 

Harrison, C., Lunzer, E. A., Tymms, P., Taylor Fitz-Gibbon, C. & Restorick, J. (2004) Use of 
ICT and its relationship with performance in examinations: a comparison of the 
ImpaCT2 project’s research findings using pupil-level, school-level and multilevel 
modelling data. Journal of Computer Assisted Learning, 20: 319-337. 

Hartley, J. (2007) Teaching, learning and new technology: a review for teachers British 
Journal of Educational Technology 38.1: 42-62. 

Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to 
achievement. London: Routledge. 

Hattie, J., Biggs, J. & Purdie, N. (1996) Effects of Learning Skills Interventions on Student 
Learning: A Meta-analysis Review of Educational Research 66.2: 99-136. 

Higgins, S., Beauchamp, G. & Miller, D. (2007) Reviewing the literature on interactive 
whiteboards Learning, Media and Technology,32.3: 213-225. 

Higgins, S., Kokotsaki, D. & Coe, R. (2012) The Sutton Trust/EEF Teaching and Learning 
Toolkit. London: Education Endowment Foundation. Retrieved from: 


http://educationendowmentfoundation.org.uk/toolkit/ 


17 


Hrastinski, S. & Keller, C. (2007) An Examination Of Research Approaches That Underlie 
Research On Educational Technology: A Review From 2000 To 2004 Journal of 
Educational Computing Research, 36.2: 175-190. 

Kanuka, H. & Kelland, J. (2008) Has e-Learning Delivered on its Promises? Expert Opinion 
on the Impact of e-Learning in Higher Education The Canadian Journal of Higher 
Education, 38.1: 45-65. 

Lei, J., & Zhao, Y. (2007). Technology uses and student achievement: A longitudinal study. 
Computers & Education, 49.2: 284-296. 

Liao, Y. C. (2007) Effects of computer-assisted instruction on students’ achievement in 
Taiwan: A meta-analysis Computers & Education 48: 216-233. 

Liao, Y.-C. & Hao, Y. (2008) Large scale studies and quantitative methods In J. Voogt & G. 
Knezek (Eds.) International handbook of information technology in primary and 
secondary education New York: Springer. 

Liu, C.-C & Kao, L.-C (2007) Do handheld devices facilitate face-to-face collaboration? 
Handheld devices with large shared display groupware to facilitate group interactions 
Journal of Computer Assisted Learning 23: 285-299. 

Luckin, R. (2008) The learner centric ecology of resources: A framework for using technology 
to scaffold learning Computers & Education 50: 449-462. 

Mabry, L. & Snow, J.Z. (2006) Laptops For High-Risk Students: Empowerment And 
Personalization in A Standards-Based Learning Environment Studies in Educational 
Evaluation 32: 289-31. 

Margolisa, J.L., Nussbaum, M., Rodriguez, P. & Rosasc, R. (2007) Methodology for 
evaluating a novel education technology: a case study of handheld video games in 
Chile Computers and Education 46.2: 174-191. 

Marzano, R. J. (1998) A Theory-Based Meta-Analysis of Research on Instruction. Aurora, 
Colorado, Mid-continent Regional Educational Laboratory. Available from: 


http://;www.peecworks.org/PEEC/PEEC Research/l01795EFA.2/Marzano%20Instructio 
n%20Meta_ An.pdf 

Mee, A. (2007) E-learning funding for schools: a policy paradox? British Journal of 
Educational Technology 38.1: 63-71. 

Moseley, D., Higgins, S., Bramald, R. Hardman, F., Miller, J., Mroz, M., Tse, H., Newton, D., 
Thompson, I., Williamson, J., Halligan, J., Bramald, S., Newton, L., Tymms, P. 
Henderson, B. & Stout, J. (1999) Ways Forward with ICT: Effective Pedagogy using 
Information and Communications Technology in Literacy and Numeracy in Primary 
Schools Newcastle upon Tyne: University of Newcastle upon Tyne. Retrieved from: 
http:/Awww.leeds.ac.uk/educol/documents/00001369.htm 

Naismith, L., Lonsdale, P., Vavoula, G. & Sharples, M. (2004) Literature Review in Mobile 
Technologies and Learning Bristol: Futurelab. Retrieved from: 
http:/Awww2.futurelab.org.uk/resources/documents/lit_reviews/Mobile Review.pdf 

OECD (2006) Are Students Ready for a Technology-Rich World? What PISA Studies Tell Us 
Paris: OECD. Retrieved from: 
http:/Awww.oecd.org/education/preschoolandschool/programmeforinternationalstudenta 


ssessmentpisa/35995145.pdf 
Oppenheimer, T. (2003) The Flickering Mind: the false promise of technology in the 


classroom and how education can be saved. New York: Random House. 

Parr, J. M. & Fung (2000) A review of the literature on computer-assisted learning, particularly 
integrated learning systems, and outcomes with respect to literacy and numeracy 
Auckland: University of Auckland. Retrieved from: 
http://www.educationcounts.govt.nz/publications/ict/5927 

Passey, D., & Higgins, S. (2011). Learning platforms and learning outcomes-—insights from 
research. Learning, Media and Technology, 36(4), 329-333. 

Penuel, W. R. (2006) Implementation and Effects of One-to-One Computing Initiatives: A 
Research Synthesis Journal of Research on Technology in Education, 38.3 p329-348. 

Ragin, C. (2000) Fuzzy Set Social Science Chicago: University of Chicago Press 

Raudenbush, S.W. (2003). Designing Field Trials of Educational Innovations. \nvited speaker 
at the DRDC Conceptualizing Scale-Up: Multidisciplinary Perspectives Conference, 
Washington D.C. Retrieved from: http://www.ssicentral.com/hlm/techdocs/DRDC.pdf 

Raudenbush, S.W. (2008) Advancing Educational Policy by Advancing Research on 
Instruction American Educational Research Journal 45; 206. 


18 


Richards, G., Lin, A., Eap, T. & Sehboub, Z. (2008) Where Do They Go? Internet Search 
Strategies in Grade Five Laptop Classrooms. |n Proceedings of World Conference on 
Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 4382-4387). 
Chesapeake, VA: AACE. 

Rogers, E. M. (2003). Diffusion of Innovations, 5th ed. New York, NY: Free Press. 

Schacter, J. & C. Fagnano. (1999) Does Computer Technology Improve Student Learning 
and Achievement? How, When, and under What Conditions? Journal of Educational 
Computing Research, 20: 329-43. 

Sipe, T. & Curlette, W.L. (1997) A Meta-Synthesis Of Factors Related To Educational 
Achievement: A Methodological Approach To Summarizing And Synthesizing Meta- 
Analyses /nternational Journal of Educational Research 25. 7.: 583-698. 

Silvernail, D.L. & Gritter, A.K. (2007) Maine's Middle Schoo! Laptop Program: Creating Better 
Writers - Research Brief Gorham, Me: Maine Education Policy Research Institute 
Available at: http://www.usm.maine.edu/cepare/Impact_on Student Writing Brief.pdf 
(accessed 29/10/08). 

Slay, H., Sieborger, |. & Hodgkinson-Williams, C. (2008) Interactive whiteboards: Real beauty 
or just lipstick? Computers & Education 51: 1321-1341. 

Somekh, B. (2007) Pedagogy and Learning with ICT: Researching the Art of Innovation, 
London: Routledge. 

Somekh, B. Underwood, J., Convery, A., Dillon, G., Jarvis , J., Lewin, C., Mavers , D., Saxon, 
D.,, Sing, S., Steadman, S., Twining, P. & Woodrow, D. (2007) Evaluation of the ICT 
Test Bed project: final report, Becta. Retrieved from: 
http://dera.ioe.ac.uk/1598/1/becta_ 2004 icttestbed annualreport_report.pdf 

Steffens, K. (2008) Technology Enhanced Learning Environments for self-regulated learning: 
a framework for research, Technology, Pedagogy and Education,17:3: 221 — 232. 

Straker, L. M., Pollock, C. M., Zubrick, S. R. & Kurinczuk, J. J. (2005) The association 
between information and communication technology exposure and physical activity, 
musculoskeletal and visual symptoms and socio-economic status in 5-year-olds Child: 
Care, Health & Development, 32.3: 343-351. 

Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What 
Forty Years of Research Says About the Impact of Technology on Learning A Second- 
Order Meta-Analysis and Validation Study. Review of Educational Research, 81.1: 4- 
28. 

Underwood, J. (2004). Research into information and communications technologies: where 
now? Technology, Pedagogy and Education, 13, 2, 135-145. 

Underwood, J. & Dillon, G. (2004) Capturing complexity through maturity modelling, 
Technology, Pedagogy and Education,13:2:213 — 225. 

Vogel, J.J., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K. & Wright M (2006) 
Computer Gaming And Interactive Simulations For Learning: A Meta-Analysis. Journal 
of Educational Computing Research, 34.3: 229-243. 

Voogt J. & Knezek, G. (Eds.) (2008) International handbook of information technology in 
primary and secondary education New York: Springer. 

Wainer, J., Dwyer, T., Dutra, R. S., Covic, A., Magalhaes, V. B., Ferreira, L. R., Pimenta, V. 
A., & Claudio, K (2008) Too much computer and Internet use is bad for your grades, 
especially if you are young and poor: Results from the 2001 Brazilian SAEB Computers 
and Education 51: 1417-1429. 

Watson, D. (1993) The Impact Report: an evaluation of the impact of information technology 
on children’s achievements. London: Kings College. 

Weaver, G.C. (2000) An examination of the National Educational Longitudinal Study 
Database to Probe the Correlation Between Computer Use in School and Improvement 
in Test Scores. Journal of Science and Technology 9.2: 121-133. 

Wenglinsky, H. (1998). Does It Compute? The Relationship Between Educational Technology 
and Achievement in Mathematics. Princeton, NJ: Policy Information Center, Research 
Division, Educational Testing Service. 

Zhao, Y. & Frank, K.A. (2003) Factors Affecting Technology Uses in Schools: An Ecological 
Perspective American Educational Research Journal 40: 807. 


19 


Appendix 1: Some contemporary myths and fallacies about digital technology use in 
education 


These personal reflections arose from discussions with the EEF team and are an attempt to 
communicate the complexity of the evidence about digital technologies and learning. They 
are included here to summarise in a less formal way what | see as the key messages in this 
field. 

Steve Higgins 


Myth 1: New technologies are being developed all the time, the past history of the 
impact of technology is irrelevant to what we have now or will be available tomorrow. 
After more than fifty years of digital technology use in education this argument is now wearing 
a bit thin. We need a clear rationale for why we think the introduction of (yet another) new 
technology will be more effective than the last one. The introduction of technology has 
consistently been shown to improve learning, the trouble is most things improve learning in 
schools when they are introduced, and technology is consistently just a little bit less effective 
than the average intervention. 


Myth 2: Today’s children are digital natives and the ‘net’ generation — they learn 
differently from older people. 

There are two issues with this myth. First, there is no evidence the human brain has evolved 
in the last 50 years, so our learning capacity remains as it was before digital technologies 
became so prevalent. It may be that young people have learned to focus their attention 
differently, but their cognitive capabilities are fundamentally the same as 30 years ago. 
Second, just because young people have grown up with technology it does not mean they are 
experts in its use for their own learning. Being an expert at playing Halo 5 requires different 
skills and knowledge from having an active Facebook account. Most young people are fluent 
in their use of some technologies, but none are expert at all of them. 


Myth 3: Learning has changed now we have access to knowledge through the internet, 
today’s children don’t need to know stuff, they just need to know where to find it. 

The web has certainly changed access to information, but it this only becomes knowledge 
when it is used for a purpose. When this requires understanding and judgement, information 
alone is insufficient. Googling is great for answers to a pub quiz, but would you trust your 
doctor if she was only using Wikipedia? To be an expert in a field you also need experience 
of using the information and knowledge, so that you understand where to focus your attention 
and where new information will help you in making decisions and judgements. It is important 
to recognise the relevance or importance of different pieces of information. Easy access to 
information can help, but it is no substitute for experience, understanding and expertise. 


Myth 4: Students are motivated by technology so they must learn better when they use 
it. 

It is certainly true that most young people do enjoy using technology in schools to support 
their learning. However, the assumption that any increased motivation and engagement will 
automatically lead to better learning is false. It is possible that increased engagement or 
motivation may help increase the time learners spend on learning activities, or the intensity 
with which they concentrate or their commitment and determination to complete a task. 
However, it is only when this engagement can be harnessed for learning that there will be any 
academic benefit. There is another caveat here as the motivation in school may be partly 
because using technology is either novel in school, or simply a change from what they usually 
experience. It may not be the case that this motivation will be sustained over time. 


Myth 5: The Everest Fallacy: we must use technology because it is there! 

We should use some of the wide range of digital technologies that are available to us to 
support learning and teaching in schools, but this should be where they improve aspects of 
teaching and learning and help to prepare children and young people for their lives after 
school. The curriculum and the way in which pupils work and are assessed should reflect the 
society and culture they are preparing pupils to be a part of when they leave formal 
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education. However the challenge is knowing which technology is the best to choose for use 
in schools and for what purposes and learning outcomes they should be employed. 


Myth 6: The “More is Better” Fallacy 

Enthusiasts assume that if a little technology is a good thing then a lot will be much better. 
The evidence does not support this assumption, for two reasons. First, large-scale 
international studies of very high use of technology — e.g. pupils using the internet more than 
four hours per day — do not show with better learning. Second, the effect of technology and 
length of interventions where more is clearly not always better! This suggests that there is an 
optimum level of technology which can support learning, too little and you don’t see the 
benefit, too much and the gains decline. A better notion might be the Goldilocks effect: it is 
about getting the amount of technology, and learners’ access to it “just right”! 


21 


Appendix 2: Summary table of Meta-analyses of the Impact of Computer and Digital Technologies on Attainment Published between 2000 and 2012 


Table 1: Meta-analyses of the Im 


act of Computer and Digital Technologies on Attainment published between 2000 and 2012 


Title Overall ES Abstract Moderator variables 
Bayraktar 0.273 This meta-analysis investigated how effective The results of this analysis also indicated that all variables 
2001 computer-assisted instruction (CAI) is on except educational level were related to effect size. The 


A Meta-analysis of 
the Effectiveness of 
Computer-Assisted 
Instruction in Science 
Education 


student achievement in secondary and college 
science education when compared to traditional 
instruction. An overall effect size of 0.273 was 
calculated from 42 studies yielding 108 effect 
sizes, suggesting that a typical student moved 
from the 50th percentile to the 62nd percentile 
in science when CAI was used. The results of 
the study also indicated that some study 
characteristics such as student-to-computer 
ratio, CAl mode, and duration of treatment were 
significantly related to the effectiveness of CAI. 


strongest relationships were found for the following 
variables: length of treatment, student-to-computer 
ratio, and publication year. Effect sizes did not vary by 
publication status and educational level. 

This study detected a significant relationship between CAI 
effectiveness and instructional role of computers. Effect 
sizes were higher (ES = 0.288) when computers were 
used as a supplement to the regular instruction and lower 
when the computer entirely replaced the regular 
instruction, {ES = 0.178). This finding was consistent with 
the previous meta-analyses (Kulik et al., 1983; Liao, 1998) 
suggesting that using the computer as a supplement to 
regular instruction should be the preferred choice instead 
of using it as a replacement. 

This meta-analysis indicated that there were no 
significant effect size differences in different school 
levels. This result supports the meta-analysis conducted 
by Flinn & Gravat (1995) reporting an effect size of 0.26 
standard deviations for elementary grades, an effect size 
of 0.20 standard deviations for secondary grades, and an 
effect size of 0.20 standard deviations for college. 
However, this finding is not consistent with the majority of 
meta-analyses (Bangert-Drowns, 1985; Burns &: 
Bozeman, 1981; Liao, 1998; Roblyer, 1989) that report 
significant effect size differences for different school levels. 
The results of this study indicated that the length of the 
treatment was strongly related to the effectiveness of CAI 
for teaching science. CAI was especially effective when 
the duration of treatment was limited to four weeks or 
less. The average effect of CAI in such studies was 0.404 
standard deviations. In studies where treatment continued 
longer than four weeks, the effects were less clear (ES = 
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0.218). A similar relationship between length of treatment 
and study outcome has been reported in previous meta- 
analyses. Kulik et al. (1983), for example, reported an 
effect size of 0.56 for 4 weeks or less, 0.30 for 5-8 weeks, 
and 0.20 for more than 8 weeks. 

This study concluded that the results found in ERIC 
documents were more positive (ES = 0.337) than results 
found in journal articles (ES = 0.293) and dissertations (ES 
= 0.229). 


Blok et al. 0.254 How effective are computer-assisted instruction | We found two study characteristics to be related to study 
2002 SE 0.056 (CAI) programs in supporting beginning outcomes. Effect sizes were higher when (a) the 
Computer-Assisted SD 0.288 readers? This article reviews 42 studies experimental group displayed an advantage at the pretest, 
Instruction in Support published from 1990 onward, comprising a total | and (b) the language of instruction was English. The 
of Beginning Reading of 75 experimental comparisons. The corrected | effects of these two predictors reduced the variability of the 
Instruction: A Review overall effect size estimate was d = 0.19 (+ study outcomes by a sizable 61 percent. 
0.06). Effect sizes were found to depend on two | Several other study characteristics appeared not to be 
study characteristics: the effect size at the time | related to study outcomes. Among these were design 
of pre-testing and the language of instruction characteristics (subject assignment, size of experimental 
(English or other). These two variables group, type of post-test score), population characteristics 
accounted for 61 percent of the variability in (regular or dyslexic readers, mean age of students), and 
effect sizes. Although an effect size of d = 0.2 treatment characteristics (type of experimental program, 
shows little promise, caution is needed because | program length, program duration). 
of the poor quality of many studies. 
Camnalbur & 1.05 (d) Studies focusing on the effectiveness of 78 studies 
Erdogan Cl 0.91 to 1.19 computer-assisted instruction have been 
2008 Random effects growing recently in Turkey. In this research, 


A Meta Analysis on 
the Effectiveness of 
Computer-Assisted 
Instruction: Turkey 

Sample 


quantitative studies comparing the 
effectiveness of computer- assisted instruction 
to traditional teaching method and conducted 
between 1998 and 2007 are studied by meta 
analysis. Seventy eight studies that have 
eligible data were combined with meta 
analytical methods by coding protocol from the 
422 master’s and doctoral degree and 124 
articles. As a result for the study, the effect size 
of computer-assisted instruction method for 
academic achievement calculated 1.048. This is 
large scale according to Thalheimer and Cook, 
large and Cohen, Welkowitz and Ewen (2000). 
Recommendations were made based on the 
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results of the study. 


A Meta Analysis: The 
Effectiveness Of The 
Use Of Mobile 
Computers On The 
Attitude And 
Academic Outcomes 
Of K-12 Students 


0.43 
(unweighted mean) 


Statistical meta analyses performed for this 
study included 32 primary studies conducted 
between 1993-2005. Two independent meta 
analyses were conducted regarding student 
attitudes and academic outcomes. The overall 
meta analysis mean by author was .23, 
indicating that student use of mobile computers 
had a small and positive effect on student 
attitudes and academic outcomes. The 
consistent pattern of positive effect size results 
indicated that student use of mobile computers 
was effective in improving student attitudes and 
academic outcomes. The small number of 
samples in the independent meta analyses 
suggests a need for further research regarding 
mobile computers. 


21 studies with data on academic outcomes 

The highest effect size for knowledge of computers and 
the Internet (.58) is moderate and positive. Academic tests 
and subject areas (.44) and Higher Order Thinking Skills 
(.26). The average academic outcome effect size mean 
(.43) is small and positive. 

The negative correlation (-.56) suggests that pre- 
experimental designs are more likely to obtain higher effect 
sizes than quasi-experimental designs. 

Duration of Study: Less than a year .39; More than a year 
but less than two years .22; More than two years .15 


Cheung & Slavin 
2011 

The Effectiveness of 
Educational 
Technology 
Applications for 
Enhancing 
Mathematics 
Achievement in K-12 
Classrooms: A Meta- 
Analysis 


0.15 
Random effects 


No abstract provided 

A total of 75 qualifying studies were included in 
our final analysis with a total sample size of 
56,886 K-12 students: 45 elementary studies 
(N=31,555) and 30 secondary studies 
(N=25,331). The overall weighted effect size is 
+0.15. 

Types of intervention. With regards to 
intervention types, the studies were divided into 
three major categories: Computer-Managed 
Learning (CML) (N=7), Comprehensive Models 
(N=8), and Supplemental CAI Technology 
(N=37). Over 70% of all studies fell into the 
supplemental program category, which consists 
of individualized computer-assisted instruction 
(CAI). These supplemental CAI programs, such 
as Jostens, PLATO, Larson Pre-Algebra, and 
SRA Drill and Practice, provide additional 
instruction at students’ assessed levels of need 
to supplement traditional classroom instruction. 
Computer-managed learning systems included 
only Accelerated Math, which uses computers 
to assess students’ mathematics levels, assign 


A marginally significant between-group effect (QB =5.58, 
df=2, p<0.06) was found, indicating some variation among 
the three programs. The 37 supplemental technology 
programs produced the largest effect size, +0.18, and 
the seven computer-managed learning programs and the 
eight comprehensive models produced similar small effect 
sizes of +0.08 and +0.06, respectively. 

The effect sizes for low, medium, and high intensity were 
+0.03, +0.20, and +0.13, respectively. In general, 
programs that were used more than 30 minutes a week 
had a bigger effect than those that were used less than 
30 minutes a week. 

The average effect size of studies with a high level of 
implementation (ES=+0.26) was significantly greater 
than those of low and medium levels of implementation 
(ES=+0.12). 

The effect sizes for low and high SES were +0.12 and 
+0.23, respectively. 

The difference between elementary studies (ES=+0.17) 
and secondary studies (ES=+0.13) was not statistically 
different. 

No publication bias was found. 

No trend toward more positive results in recent years. The 
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mathematics materials at appropriate levels, 
score tests on this material, and chart students’ 
progress. One of the main functions of the 
computer in Accelerated Math is clerical 
(Niemiec et al., 1987). Comprehensive models, 
such as Cognitive Tutor and | Can Learn, use 
computer assisted instruction along with non- 
computer activities as the students’ core 
approach to mathematics. 


mean effect sizes for studies in the 80s, 90s, and after 
2000 were +0.23, +0.15, and +0.12, respectively. 

The mean effect size for quasi-experimental studies was 
+0.19, twice the size of that for randomized studies 
(+0.10). 

The mean effect size for the 30 small studies (ES=+0.26) 
was about twice that of large studies (ES=+0.12, p<0.01). 
Large randomized studies had an effect size of +0.08, 
whereas small randomized studies had an effect size that 
was twice as large (ES=+0.17). 


Christmann & Badgett 
2003 

A Meta-Analytic 
Comparison of the 
Effects of Computer- 
Assisted Instruction 
on Elementary 
Students’ 

Academic 
Achievement 


0.34 
(unweighted mean) 


This meta-analysis compared the academic 
achievement of elementary students who 
received either traditional instruction or 
traditional instruction supplemented with CAI. 
From the 68 effect sizes, an overall mean effect 
size of 0.342 was calculated, indicating that, on 
average, students receiving traditional 
instruction supplemented with CAI attained 
higher academic achievement than did 63.31% 
of those receiving only traditional instruction. 
However, a -0.463 correlation between effect 
size and years indicates that the effect of CAI 
on academic achievement has declined 
between the years 1969 and 1998. 


39 studies 


“meta-analysis is a method of reexamining existing 
research; it is not a forecaster of prospective 
developments” p 100 


Goldberg et al. 

2003 

The Effect of 
Computers on 
Student Writing:- A 
Meta-Analysis of 
Studies from 1992 to 
2002 


0.50 (quantity) 
0.41 (quality) 


(Hedges’ g) 


Meta-analyses were performed including 26 
studies conducted between 1992-2002 focused 
on the comparison between k-12 students 
writing with computers vs. paper-and-pencil. 
Significant mean effect sizes in favor of 
computers were found for quantity of writing 
(d=.50, n=14) and quality of writing (d=.41, 
n=15). Studies focused on revision behaviors 
between these two writing conditions (n=6) 
revealed mixed results. Other studies collected 
for the meta-analysis which did not meet the 
statistical criteria were also reviewed briefly. 
These articles (n=35) indicate that the writing 
process is more collaborative, iterative, and 
social in computer classrooms as compared 
with paper-and-pencil environments. For 


As described above, regression analyses were performed 
to explore factors that may influence the effect of word 
processing on the quantity of student writing. These 
analyses indicated that student supports (i.e., keyboard 
training, technical assistance, teacher feedback, and peer 
editing) were not significant factors affecting the quantity of 
student writing. Similarly, student characteristics (i.e., 
keyboard experience prior to the study, student 
achievement level, school setting, and grade level) also 
were not significant factors affecting the quantity of student 
writing, although grade level did approach statistical 
significance. Finally, the study characteristics (i.e., 
publication type, presence of control group, pre-post 
design, length of study) were not related to the effect of 
word processing on the quantity of student writing. 
Recognizing that studies that lasted for less than six weeks 
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educational leaders questioning whether 
computers should be used to help students 
develop writing skills, the results of the meta- 
analyses suggest that, on average, students 
who use computers when learning to write are 
not only more engaged and motivated in their 
writing, but they produce written work that is of 
greater length and higher quality. 


may not provide enough time for the use of word 
processors to impact student writing, a separate set of 
regression analyses were performed for the sub-set of 
studies that lasted more than six weeks. For this sub-set of 
studies, a significant relationship between school level and 
effect size was found. On average, effect sizes were 
larger for studies that focused on middle and high 
school students as compared to elementary students. 
All other factors remained insignificant. This suggests that 
the relationship between school level and quality of writing 
occurred regardless of the length of study. 


Graham & Perin 
2007 

A Meta-Analysis of 
Writing Instruction for 
Adolescent Students 
(see also Morphy and 
Graham, 2012) 


0.56 
(unweighted mean) 


There is considerable concern that the majority 
of adolescents do not develop the competence 
in writing they need to be successful in school, 
the workplace, or their personal lives. A 
common explanation for why youngsters do not 
write well is that schools do not do a good job of 
teaching this complex skill. In an effort to 
identify effective instructional practices for 
teaching writing to adolescents, the authors 
conducted a meta-analysis of the writing 
intervention literature (Grades 4—12), focusing 
their efforts on experimental and quasi- 
experimental studies. They located 123 
documents that yielded 154 effect sizes for 
quality of writing. The authors calculated an 
average weighted effect size (presented in 
parentheses) for the following 11 interventions: 
strategy instruction (0.82), summarization 
(0.82), peer assistance 

(0.75), setting product goals (0.70), word 
processing (0.55), sentence combining (0.50), 
inquiry (0.32), prewriting activities (0.32), 
process writing approach 0.32), study of 
models (0.25), grammar instruction (— 0.32). 


The study focuses on general approaches to improve 
writing. 

18 studies of word processing approaches has an ES of 
0.56 (0.43 to 0.67) SD. No overall pooled effect for the 
meta-analysis. Effects of different approaches ranged from 
ranged from Strategy Instruction 1.03 and setting product 
goals (1.0) to Grammar approaches ( -0.22). 


Kulik 

2003 

Effects of Using 
Instructional 
Technology in 


0.30 writing quality 
0.84 WTR — in K 
0.4 WTR-Gi1 
0.25 WTR — G2+ 
0.06 reading/ILS 


This report reviews findings from controlled 
evaluations of technology applications in 
elementary and secondary schools published 
since 1990 located through computer searches 
of library databases... and summarises reviews 


Reading 

27 controlled evaluation studies on instructional technology 
and reading which focused on three major applications of 
technology to reading instruction: (a) integrated learning 
systems; (b) writing-based reading programs; and (c) 
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Elementary and 
Secondary Schools: 
What Controlled 
Evaluation Studies 
Say (a) Reading and 
writing 


0.43 —-AR 
0.30 WP on writing 
quality 


of studies published before 1990. 


reading management programs. 

Integrated learning systems: Nine controlled studies 
conducted during the last decade suggest that ILSs have 
done little to improve teaching effectiveness of reading 
programs. In each study, reading scores of children 
learning with ILSs were as high as reading scores of those 
studying in traditional classrooms, but results for ILS 
instruction were significantly better in only three of the nine 
studies. The median effect of ILS instruction in the nine 
studies was to raise students reading scores by 0.06 
standard deviations, a trivial increment. 

Writing-based reading programs: Writing to Read (WTR) is 
a program that attempts to teach young children to read by 
stimulating them to write. Twelve evaluation studies 
conducted during the past decade found that WTR effects 
were large in kindergartens (0.84), moderate in size in 
Grade 1 (0.4), and small in grades beyond Grade 1 (0.25). 
Reading management programs: Reading management 
programs, such as Accelerated Reader (AR), help 
students make book selections and then test the 
students on their understanding of what they have 
read. Results of three controlled comparisons 
suggest that AR has an effect of 0.43 standard 
deviations. 

Writing: 12 controlled studies of technology effects on 
student writing. The 12 studies fall into three categories: 
(a) word processing studies; (b) studies of computer 
writing prompts; and (c) studies of computer enrichment. 
Word processing: Four evaluation studies from the past 
decade also examined word processing effects on writing 
skills. In three out of the four studies, word processing 
produced significant positive effects on student writing 
skills. In the remaining study, however, writing with word 
processors had a significant negative effect on student 
writing skills. The median effect in the four studies was to 
increase writing skill, as measured by ratings of 
quality of their compositions, by 0.30 standard 
deviations. 

Computer writing prompts Prompting seems to be effective 
when students receive unsolicited writing prompts, but 
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prompting seems to be ineffective when students must ask 
the computer for prompts. Clearly, more research is 
needed to confirm this conclusion. 

Computer enrichment: Five out of the six studies found 
that computer enrichment helped students to improve their 
writing skills. In the remaining study, computer enrichment 
had a small, statistically significant, negative effect on 
student writing. The median effect size of computer 
enrichment programs in the six studies was an 
increase in writing scores of 0.34 standard deviations. 


Kulik 

2003 

Effects of Using 
Instructional 
Technology in 
Elementary and 
Secondary Schools: 
What Controlled 
Evaluation Studies 
Say (a) Mathematics 
and Science 


0.38* -ILS on maths 
0.59* - Computer 
tutorials in science] 
0.32* - Simulations 
in science] 

0.01* - Live labs 
(*median ES) 


This report reviews findings from controlled 
evaluations of technology applications in 
elementary and secondary schools published 
since 1990 located through computer searches 
of library databases... and summarises reviews 
of studies published before 1990. 


Mathematics and Science 

Also reviewed in this report are 36 controlled studies of 
technology effects on mathematics and science learning. 
The 36 studies covered computer applications in four 
areas: (a) integrated learning systems in mathematics; (b) 
computer tutorials; (c) computer simulations; and (d) 
microcomputer-based laboratories. 

Integrated learning systems in mathematics. 16 controlled 
studies all positive on mathematics test scores; in nine the 
ILS effect was statistically significant and educationally 
meaningful. The median ILS effect 0.38 SD. NB Lower 
effects when students do both reading and maths ILS. 
Computer tutorials. Six studies of computer tutorials in 
the natural and social sciences. In all but one of the six 
cases, the effect of computer tutoring was large enough to 
be considered both statistically significant and 
educationally meaningful. In the remaining study, the boost 
from computer tutoring was near zero. Median case 0.59 
SD. 

Computer simulations. Four of the six studies found 
positive effects on student learning, and two studies found 
negative 

effects. Median case 0.32 SD. 

Microcomputer-based laboratories (e.g. electronic sensors 
collecting data which is represented live). Seven of the 
eight studies found either small negative or small positive 
effects of MBL instruction on student learning. Median ES 
0.01. 


LeJeune 
2002 


0.34 Lower order 
outcomes 


The purpose of this study was to synthesize the 
findings from existing research on the effects of 


0.14 K-12 LOTS (0.49 College/Adult) 
0.42 K-12 HOTS 
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A meta-analysis of 
outcomes from the 
use of computer- 
simulated 
experiments in 
science education 


0.38 Higher order 
outcomes 

0.19 retention follow 
up test 


computer simulated experiments on students in 
science education. Results from 40 reports 
were integrated by the process of meta-analysis 
to examine the effect of computer-simulated 
experiments and interactive videodisc 
simulations on student achievement and 
attitudes. Findings indicated significant positive 
differences in both low-level and high-level 
achievement of students who use computer- 
simulated experiments and interactive videodisc 
simulations as compared to students who used 
more traditional learning activities. No 
significant differences in retention, student 
attitudes toward the subject, or toward the 
educational method were found. Based on the 
findings of this study, computer-simulated 
experiments and interactive videodisc 
simulations should be used to enhance 
students' learning in science, especially in 
cases where the use of traditional laboratory 
activities are expensive, dangerous, or 
impractical. 


0.39 Physical Sciences LOTS 
0.27 Biological Sciences LOTS 
0.35 Physical Sciences HOTS 
0.41 Biological Sciences HOTS 
0.46 More than one week 

0.33 Less than one week 


Lower ES for most recent studies 


Li & Ma 

2010 

A Meta-Analysis of 
the Effects of 
Computer 
Technology on 
School Students’ 
Mathematics 
Learning 


0.71 


This study examines the impact of computer 
technology (CT) on mathematics education in 
K-12 classrooms through a systematic review of 
existing literature. A meta-analysis of 85 
independent effect sizes extracted from 46 
primary studies involving a total of 36,793 
learners indicated statistically significant 
positive effects of CT on mathematics 
achievement. In addition, several 
characteristics of primary studies were 
identified as having effects. For example, CT 
showed advantage in promoting mathematics 
achievement of elementary over secondary 
school students. As well, CT showed larger 
effects on the mathematics achievement of 
special need students than that of general 
education students, the positive effect of CT 
was greater when combined with a 


Four characteristics of the studies remained statistically 
significant collectively. Two of them indicated large effects. 
With other statistically significant variables controlled, 
special education status showed a magnitude of 1.02 
SD in favor of applying technology to special need 
students over general education students, and method of 
teaching showed a magnitude of 0.79 SD in favor of 
using technology in school settings where teachers 
practiced constructivist approach to teaching over 
school settings where teachers practiced traditional 
approach to teaching. Meanwhile, two characteristics 
indicated moderate and small effects of technology on 
mathematics achievement. Year of publication showed a 
moderate magnitude of 0.32 SD in favor of publications 
before the turn of the century (before 1999) over 
publications after the turn of the century (after 1999), with 
other statistically significant variables controlled. Type (or 
level) of education showed a small magnitude of 0.22 SD 
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constructivist approach to teaching than with a 
traditional approach to teaching, and studies 
that used non-standardized tests as measures 
of mathematics achievement reported larger 
effects of CT than studies that used 
standardized tests. The weighted least squares 
univariate and multiple regression analyses 
indicated that mathematics achievement could 
be accounted for by a few technology, 
implementation and learner characteristics in 
the studies. 


in favor of using technology at the elementary school 
level over second school level, with other statistically 
significant variables controlled. 


Liao 0.55 A meta-analysis was performed to synthesize Significantly higher effect sizes for underpowered studies 
2005 existing research comparing the effects of (1.39 versus 0.48) and for no comparison group studies 
Effects of computer- computer-assisted instruction (CAI) versus 1.30 (versus 0.43 with TI controls), but all pre-post 
assisted instruction traditional instruction (Tl) on students’ controlled studies ES is 0.56. Other non sig, differences 
on students’ achievement in Taiwan. 52 studies were highest effect sizes for 4-18 hours of treatment (1.18). Use 
achievement in located from our sources, and their quantitative | as supplement to (0.54) higher than replacement for 
Taiwan: A meta- data was transformed into effect size (ES). The | (0.53). Small (<5) group (0.96) higher than individual 
analysis overall grand mean of the study-weighted ES (0.56). Subjects: maths (0.29), language (0.66), science 

for all 52 studies was 0.55. The results suggest | (0.49) computer (0.76). Age Grades 1-6 0.41, 7-9 0.85, 10- 

that CAI is more effective than TI in Taiwan. In 12 0.23, College 0.82. Publication type: Journal article 

addition, two of the seventeen variables 0.45, dissertation/thesis 0.53, NSC project 0.67 

selected for this study (i.e., statistical power, 

and comparison group) had a statistically 

significant impact on the mean ES. The results 

from this study suggest that the effects of CAI in 

instruction are positive over Tl. The results also 

shed light on the debate of learning from media 

between Clark and Kozma. 
Lou et al 0.16 This study quantitatively synthesized the The overall effect of social context on individual 
2001 empirical research on the effects of social achievement was based on 178 independent effect sizes 


Small Group and 
Individual Learning 
with Technology: A 
Meta-Analysis 


context (i.e., small group versus individual 
learning) when students learn using computer 
technology. In total, 486 independent findings 
were extracted from 122 studies involving 
11,317 learners. The results indicate that, on 
average, small group learning had significantly 
more positive effects than individual learning on 
student individual achievement (mean ES = 
+0.15), group task performance (mean ES = 


extracted from 100 studies. The mean weighted effect size 
(d+) was +0.16 (95% Cl is +0.12 to +0.20; and QT = 
341.95, df = 177, p < .05) before outlier procedures. 
Individual effect sizes ranged from -1.14 to +3.37, with 105 
effect sizes above zero favoring learning in groups, 15 
effect sizes equal to zero, and 58 effect sizes below zero 
favoring individual learning. Fifteen outliers with 
standardized residuals larger than +2.00 were identified. 
After outlier procedures, the mean effect size was +0.15 
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+0.31), and several process and affective 
outcomes. However, findings on both individual 
achievement and group task performance were 
significantly heterogeneous. Through weighted 
least squares univariate and multiple regression 
analyses, we found that variability in each of the 
two cognitive outcomes could be accounted for 
by a few technology, task, grouping, and 
learner characteristics in the studies. 

The results of Hierarchical Regression Model 
development indicate that the effects of small 
group learning with CT on individual 
achievement were significantly larger when: (a) 
students had group work experience or specific 
instruction for group work rather than when no 
such experience or instruction was reported; (b) 
cooperative group learning strategies were 
employed rather than general encouragement 
only or individual learning strategies were 
employed; (c) programs involved tutorials or 
practice or programming languages rather than 
exploratory environments or as tools for other 
tasks; (d) subjects involved social sciences or 
computer skills rather than mathematics, 
science, reading, and language arts; (e) 
students were relatively low in ability rather than 
medium or high in ability; and (f) studies were 
published in journals rather than not published. 
When all the positive conditions were present, 
students learning in small groups could achieve 
0.66 standard deviation more than those 
learning individually. When none of the positive 
conditions were present, students learning 
individually could learn 0.20 standard deviation 
more than those learning in groups. 


(95% confidence interval is +0.11 to +0.19). The results 
indicate that, on average, there was a small but 
significantly positive effect of small group learning on 
student achievement as measured by individually 
administered immediate or delayed post-tests. 

Effect sizes were significantly larger when students 
were learning with tutor programs (d+ = +0.20) or 
programming languages (d+ = +0.22) than when using 
exploratory or tool programs (d+ = +0.04). 

Effect sizes greatest for low attaining learners (0.34) as 
compared with medium (0.09), high (0.24) or 
mixed(0.12). 

The effects of social context on student individual 
achievement were larger when the subjects involved were 
computer skills (d+ = +0.24), social sciences and other (d+ 
= +0.20) than when the subjects were 
math/science/language arts (d+ = +0.11). 

The effect sizes were significantly positive for both 
heterogeneous ability groups (d+ = +0.21) and 
homogeneous ability groups (d+ = +0.22). 

Effect sizes were significantly more positive when specific 
cooperative learning strategies were employed (d+ = 
+0.21) than when students were generally encouraged to 
work together (d+ = -0.04) or when students in groups 
worked under individualistic goals or when no group 
learning strategy was described in the study (d+ = +0.08), 
with the latter two means not significantly different from 
zero. 

Significantly more positive when students worked in 
pairs (d+ = +0.18) than when they worked in three to five 
member groups (d+ = +0.08). 

Type of feedback, types of tasks, task familiarity, task 
difficulty, number of sessions, session duration, grade 
level, gender, computer experience, instructional control, 
and whether achievement outcomes measured were of 
higher-order skills or lower-order skills were not found to 
be significantly related to the variability in the effects of 
social context on student individual achievement. 
Individuals appear to benefit from computer-based 
feedback but groups do better without computer-based 
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feedback when completing group tasks. 


Means et al. 

2009 

Evaluation of 
Evidence-Based 
Practices in Online 
Learning: A Meta- 
Analysis and Review 
of Online Learning 
Studies 


0.24 


A systematic search of the research literature 
from 1996 through July 2008 identified more 
than a thousand empirical studies of online 
learning. Analysts screened these studies to 
find those that (a) contrasted an online to a 
face-to-face condition, (b) measured student 
learning outcomes, (c) used a rigorous research 
design, and (d) provided adequate information 
to calculate an effect size. As a result of this 
screening, 51 independent effects were 
identified that could be subjected to meta- 
analysis. The meta-analysis found that, on 
average, students in online learning conditions 
performed better than those receiving face-to- 
face instruction. The difference between 
student outcomes for online and face-to-face 
classes—measured as the difference between 
treatment and control means, divided by the 
pooled standard deviation—was larger in those 
studies contrasting conditions that blended 
elements of online and face-to-face instruction 
with conditions taught entirely face-to-face. 
Analysts noted that these blended conditions 
often included additional learning time and 
instructional elements not received by students 
in control conditions. This finding suggests that 
the positive effects associated with blended 
learning should not be attributed to the media, 
per se. An unexpected finding was the small 
number of rigorous published studies 
contrasting online and face-to-face learning 
conditions for K-12 students. In light of this 
small corpus, caution is required in generalizing 
to the K-12 population because the results are 
derived for the most part from studies in other 
settings (e.g., medical training, higher 
education). 

Few rigorous research studies of the 
effectiveness of online learning for K-12 


Students who took all or part of their class online 
performed better, on average, than those taking the same 
course through traditional face-to-face instruction. Learning 
outcomes for students who engaged in online learning 
exceeded those of students receiving face-to-face 
instruction, with an average effect size of +0.24 
favoring online conditions. The mean difference 
between online and face-to-face conditions across the 51 
contrasts is statistically significant at the p < .01 level. 
Interpretations of this result, however, should take into 
consideration the fact that online and face-to-face 
conditions generally differed on multiple dimensions, 
including the amount of time that learners spent on task. 
The advantages observed for online learning conditions 
therefore may be the product of aspects of those treatment 
conditions other than the instructional delivery medium per 
se. 

Instruction combining online and face-to-face elements 
had a larger advantage relative to purely face-to-face 
instruction than did purely online instruction. The mean 
effect size in studies comparing blended with face-to- 
face instruction was +0.35, p < .001. This effect size is 
larger than that for studies comparing purely online and 
purely face-to-face conditions, which had an average effect 
size of +0.14, p < .05. An important issue to keep in mind 
in reviewing these findings is that many studies did not 
attempt to equate (a) all the curriculum materials, (b) 
aspects of pedagogy and (c) learning time in the treatment 
and control conditions. Indeed, some authors asserted that 
it would be impossible to have done so. Hence, the 
observed advantage for online learning in general, and 
blended learning conditions in particular, is not 
necessarily rooted in the media used per se and may 
reflect differences in content, pedagogy and learning 
time. 

Studies in which learners in the online condition spent 
more time on task (0.46) than students in the face-to-face 
condition found a greater benefit for online 
learning...compared with +0.19 for studies in which the 
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students have been published. A systematic 
search of the research literature from 1994 
through 2006 found no experimental or 
controlled quasi-experimental studies 
comparing the learning effects of online versus 
face-to-face instruction for K-12 students that 
provide sufficient data to compute an effect 
size. A subsequent search that expanded the 
time frame through July 2008 identified just five 
published studies meeting meta-analysis 
criteria. 


learners in the face-to-face condition spent as much time 
or more on task. 

Effect sizes were larger (0.42) for studies in which the 
online and face-to-face conditions varied in terms of 
curriculum materials and aspects of instructional 
approach in addition to the medium of instruction 
rather than those which replicated the instruction and 
curriculum (0.20). 

The meta-analysis did not find differences in average 
effect size between studies published before 2004 (which 
might have used less sophisticated Web-based 
technologies than those available since) and studies 
published from 2004 on (possibly reflecting the more 
sophisticated graphics and animations or more complex 
instructional designs available). Nor were differences 
associated with the nature of the subject matter involved. 
Finally, the examination of the influence of study method 
variables found that effect sizes did not vary significantly 
with study sample size or with type of design. 


Moran et al. 

2008 

Technology and 
Reading Performance 
in the Middle-School 
Grades: A Meta- 
Analysis with 
Recommendations for 
Policy and Practice 


(Also Pearson, 2005, 
The Effects of 
Technology on 
Reading Performance 
in the Middle-School 
Grades: A Meta- 
Analysis with 
Recommendations for 
Policy and Practice) 


0.49 


The results of a meta-analysis of 20 research 
articles containing 89 effect sizes related to the 
use of digital tools and learning environments to 
enhance literacy acquisition for middle school 
students demonstrate that technology can have 
a positive effect on reading comprehension 
(weighted effect size of 0.489). Very little 
research has focused on the effect of 
technology on other important aspects of 
reading, such as metacognitive, affective, and 
dispositional outcomes. The evidence permits 
the conclusion that there is reason to be 
optimistic about using technology in middle- 
school literacy programs, but there is even 
greater reason to encourage the research 
community to redouble its efforts to investigate 
and understand the impact of digital learning 
environments on students in this age range and 
to broaden the scope of the interventions and 
outcomes studied. 


1. The effect sizes were greater for interventions aimed at 
general populations than those with specific needs (i.e., 
students who are learning disabled or struggling readers). 
For the 57 effect sizes reported for a general, 
undifferentiated population of middle school students, the 
mean effect size was 0.52, whereas the effect size for 
targeted populations of students (e.g., students 
classified as possessing learning disabilities or as 
struggling readers) was 0.32: this was a statistically 
reliable difference. We can only speculate about why this 
might be the case, and we surely need more evidence 
before reaching a definitive conclusion. However, issues of 
engagement and appropriate levels of support and 
feedback suggest themselves as reasonable explanations. 
2. Standardized measures from test companies (0.30), 
were less sensitive to treatment effects than researcher- 
developed measures in several of the studies in this meta- 
analysis measures were less sensitive to treatment effects 
than experimenter-designed assessments (0.56). 

3. Sample size was a robust predictor of effect size; small 
n studies (30 or less) produced 14 effect sizes averaging 
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0.77, while large n (31 or more) studies produced 75 effect 
sizes with a mean of 0.38, Q D 3:24; p < 0:20. Studies with 
smaller sample sizes were much more likely to achieve 
substantial effects than those with larger sample sizes. 
This counter-intuitive finding is puzzling because of what 
we know about the increase in statistical power that comes 
with larger experimental samples. On the other hand, there 
may be a trade-off between statistical power and 
experimental precision; that is, it may be easier for 
researchers to maintain a high degree of fidelity to 
treatment in smaller studies because of the greater 
manageability prospects. 

4. Technologies that were created by a research team 
(1.20) had a much larger effect size than those 
technologies either adapted from the commercial 
market (0.28) or those that merely used the technology 
as a delivery system (0.34). This finding may be related 
to the fact that those technologies created by researchers 
tended to have a clear theoretical focus that was matched 
by the assessments employed by the team. In short, 
alignment between intention and outcome measure may 
be the operative variable behind this robust finding. 

5. Studies that used some sort of correlated design 
(pretests used as covariates for posttest or repeated 
measures designs in which the same subjects cycle 
through different interventions) are more likely to find 
reliable differences between interventions than are 
independent group designs. 

6. Effect sizes in studies lasting two to four weeks 
(0.55) were larger than those in studies lasting less 
than a week (0.48) but much larger than those from 
studies lasting five or more weeks (0.34). 


Morphy & Graham 
2012 
Word-processing 
programs and weaker 
writers/ readers: a 
meta-analysis of 
research findings 


0.52 - writing quality 

0.48 — length 

0.66 - development/ 
organization of 

text 

0.61 mechanical 
correctness 


Since its advent word processing has become a 
common writing tool, providing potential 
advantages over writing by hand. Word 
processors permit easy revision, produce 
legible characters quickly, and may provide 
additional supports (e.g., spellcheckers, speech 
recognition). Such advantages should remedy 
common difficulties among weaker 


While basic word processing impacted writing quality 
positively, neither the addition of external instructional 
support (WP+; A = -0.28; p = 0.123) nor the use of voice 
recognition (VR; A = -0.20; p = 0.26) differed significantly 
from basic word processors alone. Conversely, three 
interventions which added internal support to the word 
processor (WP++) were associated with considerable 
gains in writing quality (Ad = 0.91; p = 0.002) when 
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writers/readers in grades 1-12. Based on 27 
studies with weaker writers, 20 of which were 
not considered in prior reviews, findings from 
this meta-analysis support this proposition. 
From 77 independent effects, the following 
average effects were greater than zero: writing 
quality (d = 0.52), length (d = 0.48), 
development/organization of text (d = 0.66), 
mechanical correctness (d = 0.61), motivation 
to write (d = 1.42), and preferring word 
processing over writing by hand (d = 0.64). 
Especially powerful writing quality effects were 
associated with word processing programs that 
provided text quality feedback or prompted 
planning, drafting, or revising (d = 1.46), 
although this observation was based on a 
limited number of studies (n = 3). 


compared to P&P. 

Random assignment and rater blinding were both 
associated with larger writing quality effects, but only 
random assignment proved statistically significant. 


Onuoha 

2007 

Meta-Analysis Of The 
Effectiveness Of 
Computer-Based 
Laboratory Versus 
Traditional Hands-On 
Laboratory In College 
And Pre-College 
Science Instructions 


0.26 


The purpose of this research study was to 
determine the overall effectiveness of 
computer-based laboratory compared with the 
traditional hands-on laboratory for improving 
students’ science academic achievement and 
attitudes towards science subjects at the 
college and pre-college levels of education in 
the United States. Meta-analysis was used to 
synthesis the findings from 38 primary research 
studies conducted and/or reported in the United 
States between 1996 and 2006 that compared 
the effectiveness of computer-based laboratory 
with the traditional hands-on laboratory on 
measures related to science academic 
achievements and attitudes towards science 
subjects. The 38 primary research studies, with 
total subjects of 3,824 generated a total of 67 
weighted individual effect sizes that were used 
in this meta-analysis. The study found that 
computer-based laboratory had small positive 
effect sizes over the traditional hands-on 
laboratory (ES = +0.26) on measures related to 
students’ science academic achievements and 


A total of 35 independent primary studies with a total of 
3,284 subjects met the inclusion criteria to answer the 
primary research question. The 35 primary studies 
generated 35 weighted effect sizes (w), one effect size (d) 
for each primary study. The individual effect sizes ranged 
from a low negative effect size of -0.38, to a large positive 
effect size of + 1.12. The overall mean effect size (ES), 
calculated at 95% confidence interval was + 0.26 standard 
deviation units. Twelve primary studies representing 34% 
of the studies analyzed reported negative effect sizes. the 
overall mean effect size (ES) for physical science subjects 
(physics and chemistry) was +0.34 standard deviation 
units, while biological science was +0.17 standard 
deviation units. The effect on Pre-College studies was 
+0.24 standard deviation units compared to +0.21 obtained 
for College level studies. 

ES for science attainment in studies between 1996 and 
2000 +0.33 compared with +0.19 standard deviation units 
for studies conducted between 2001 and 2006. 
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attitudes towards science subjects (ES = 
+0.22). It was also found that computer-based 
laboratory produced more significant effects on 
physical science subjects compared to 
biological sciences (ES = +0.34, +0.17). 


Rosen & Salomon 
2007 

The Differential 
Learning 
Achievements Of 
Constructivist 
Technology-Intensive 
Learning 
Environments As 
Compared With 
Traditional Ones: A 
Meta-Analysis 


Different learning environments provide 
different learning experiences and ought to 
serve different achievement goals. We 
hypothesized that constructivist learning 
environments lead to the attainment of 
achievements that are consistent with the 
experiences that such settings provide and that 
more traditional settings lead to the attainments 
of other kinds of achievement in accordance 
with the experiences they provide. A meta- 
analytic study was carried out on 32 
methodologically-appropriate experiments in 
which these 2 settings were compared. Results 
supported 1 of our hypotheses showing that 
overall constructivist learning environments are 
more effective than traditional ones (ES = .460) 
and that their superiority increases when tested 
against constructivist-appropriate measures 
(ES = .902). However, contrary to expectations, 
traditional settings did not differ from 
constructivist ones when traditionally- 
appropriate measures were used. A number of 
possible interpretations are offered among them 
the possibility that traditional settings have 
come to incorporate some constructivist 
elements. This possibility is supported by other 
findings of ours such as smaller effect sizes for 
more recent studies and for longer lasting 
periods of instruction. 


Grade level was found to moderately affect the results— 
although the effect sizes favored CTILE regardless of 
grade level, still the effect size for grades 1-6 were 
significantly smaller than those for grades 7-9 (d+ = 
.413, d+ = 0.583 respectively, Qb = 5.29, p < .05). 
Consiructivist learning environments yielded 
significantly higher achievements than traditional ones 
when math instruction lasted for up to six weeks as 
compared with instruction that lasted for seven weeks or 
more (d+ = .686, d+ = .408 respectively, Qb = 10.76, p < 
.01). 

Also year of publication made a difference—CTILE yielded 
larger effect sizes when the studies were published 
between 1986 and 1991 than between 1992 and 2002 (d+ 
= .554, d+ = .388 respectively, Qb = 6.16, p < .05). 


Sandy-Hanson 

2006 

A meta-analysis of 
the impact of 
computer technology 
versus traditional 


0.24 
(SD 0.47; SE 0.017) 


Meta-analytical research has shown that 
computer technology can play a significant role 
in increasing positive learning outcomes of 
students. Research on this topic has resulted in 
conflicting findings on academic achievement 
and other measures of student outcomes. The 


Studies reported from 2000-2003 ES 0.09; 2004-2006 ES 
0.24. 

Grade level: Pre-K-5 — 0.49; Grade 6-8 0.07; Grade 9-12 
0.31; Multiple levels 0.23. 

Sample size <100 — 0.55; 101-200 — 0.52; >200 0.24 
Subjects: English 0.07; Science 0.51; mathematics 0.28; 
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instruction on 
students in 
Kindergarten through 
12" Grade in the 
United States. 


current meta-analysis sought to assess the 
level of differences that existed between 
students being instructed with computer 
technology versus the academic achievement 
outcomes of students instructed with traditional 
methods. Based on specified selection criteria, 
31 studies were collected and analyzed for 
homogeneity. From this original group, 23 
studies were systematically reviewed under 
standard meta-analytical procedures. According 
to Cohen's (1988) classification of effect sizes 
in the field of education, the obtained weighted 
mean effect size of .24 shows a medium 
difference. This finding indicates that students 
who are taught with technology outperform their 
peers who are taught with traditional methods 
of instruction. In addition, five secondary 
analyses were conducted on higher-order 
thinking skills, ES = .82, motivation, ES = .17, 
retention-attendance rates, ES = .16, physical 
outcomes, no data were found, and social skills, 
ES = .21. Eleven ancillary analyses were then 
conducted to assess study findings across 
various dimensions including duration of study, 
type of technology used, and grade-level 
analyzed. 


History -0.20 (one study). 

Student type: General education 0.24; Special Ed 0.60; 
Mixed 0.01. 

Test type: standardized 0.24; intelligence test 0.58; 
Teacher/researcher 0.13; GPA 0.12 

Duration: Up to 2 weeks 0.57; 3-4 weeks 0.50; 5 to 35 
weeks 0.26; 36 weeks or more 0.26 

Technology type: CAI tools 0.52; Whole environment - 
0.04; Online/distance 0.26; Hardware/software 0.50 


Seo & Bryant 

2009 

Analysis of studies of 
the effects of 
computer-assisted 
instruction on the 
mathematics 
performance of 
students with learning 
disabilities 


NPE 


The purpose of this study was to conduct a 
meta-study of computer-assisted instruction 
(CAI) studies in mathematics for students with 
learning disabilities (LD) focusing on examining 
the effects of CAI on the mathematics 
performance of students with LD. This study 
examined a total of 11 mathematics CAI 
studies, which met the study selection criterion, 
for students with LD at the elementary and 
secondary levels and analyzed them in terms of 
their comparability and effect sizes. Overall, this 
study found that those CAI studies did not show 
conclusive effectiveness with relatively large 
effect sizes. The methodological problems in 


CAI versus teacher instruction: The four group-design 
studies were associated with a small to medium effect size 
(d = 0.09, 0.33, 0.45, and 0.75). 

Comparison of CAI types: The two group-design studies 
compared the effectiveness of drill and practice CAI with 
game CAI for enhancing the addition skills of students with 
LD. Results of these studies demonstrated contradictory 
findings (d = 0.71 and -0.47 for drill and practice CAI). 
Enhanced CAI: The two group-design studies were related 
with either a small or large effect size (d = 0.87 and 0.30). 
The purpose of this study was to conduct a meta-study of 
mathematics CAI studies for students with LD. The 11 
mathematics CAI studies were selected and examined 
their effectiveness for enhancing the mathematics 
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the CAI studies limit an accurate validation of 
the CAI’s effectiveness. Implications for future 
mathematics CAI studies were discussed. 


performance of students with LD with their effect sizes. 
The results of this study found that the CAI studies in 
mathematics did not show conclusive effectiveness for the 
mathematics performance of students with LD with 
relatively large effect sizes. 


A Meta-Analytic 
Investigation Into The 
Efficacy Of Fast 
ForWord Intervention 
On Improving 
Academic 
Performance 


0.35 (mean ES) 
0.22 -—on 
Standardised 
reading tests 


There has been contradictory evidence 
concerning the validity of auditory temporal 
processing deficits as a cause for reading and 
language problems. In spite of the controversy, 
Merzenich and Tallal helped develop a popular 
computer-based intervention, Fast ForWord 
(Scientific Learning Corporation [SLC], 2006). 
Although a variety of studies have examined 
the effectiveness of FFW on academic 
performance, the findings have been 
inconsistent, creating the need to quantitatively 
synthesize findings of experimental studies on 
Fast ForWord. Thirty-one studies met the 
stipulated inclusion criteria, which generated 
163 effect sizes aggregated across academic 
skills (e.g., reading, language, phonological 
processing). The overall mean effect size was 
in the small to medium range, and no particular 
reading, language, or phonological processing 
skill appeared to be significantly more 
responsive to FFW than another skill. All mean 
effect sizes were associated with sizable 
variability, often equal to or exceeding effect 
size, which decreased the confidence one could 
place in the "true" effect of FFW. Aggregations 
were also made across moderator variables 
(e.g., grade, ethnicity, diagnostic category). 
This paper provides supporting evidence on the 
need for the study, a review of the related 
auditory temporal processing literature, and the 
purpose, procedure, and findings of the meta- 
analysis. 


Word recognition 0.28; Comprehension 0.28; Fluency 
0.57; Vocabulary 0.37; Standardised reading tests 0.22 
Spelling 0.21 

Elementary grades 0.43 

Special Education students 0.52 

Outside school hours 0.48 / 0.28 regular school day 
SLC sponsored studies 0.43 / independent studies 0.20 


Soe et al. 
2000 
The effect of 


0.26 
(r= 0.1316) 


Whether computer-assisted instruction (CAI) 
can improve reading achievement of students 
has been a crucial question addressed by 


The overall finding of this meta-analysis is that computer- 
assisted instruction has a positive impact on reading 
achievement. However, there is a wide range in the foci, 
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computer-assisted 
instruction (CAI) on 
reading achievement 


studies in the past. This meta-analysis reviewed 
17 research studies based on students K-12 
and revealed that CAI does have a positive 
effect on reading achievement. Although the 
effects of CAI in 17 studies were not 
homogeneous, there seems to be no particular 
study characteristic that might have caused the 
heterogeneity. 


procedures, materials, and findings among the studies 
included in this meta-analysis. In some cases, a scarcity of 
acceptable studies was evident in many categories. 
Therefore, the results given here must be interpreted with 
caution until a greater number of similar studies with 
similar reporting styles is available to confirm or refute the 
findings. 

Lack of sufficient numbers of studies in key areas could 
perhaps be the greatest barrier to the systematic 
assessment of the impact of CAI on the teaching of 
reading. Findings indicate that computer applications can 
play a significant role in teaching and learning. However, 
the precise nature of that role still needs to be researched 
with greater depth and precision. 


Strong et al. 

2011 

A systematic meta- 
analytic review of 
evidence for the 
effectiveness of the 
‘Fast ForWord’ 
language intervention 
program 


Fast ForWord is a suite of computer-based 
language intervention programs designed to 
improve children’s reading and oral language 
skills. The programs are based on the 
hypothesis that oral 

language difficulties often arise from a rapid 
auditory temporal processing deficit that 
compromises the development of phonological 
representations. Methods: A systematic review 
was designed, undertaken and reported using 
items from the PRISMA statement. A literature 
search was conducted using the terms ‘Fast 
ForWord’ ‘Fast For Word’ ‘Fastforword’ with no 
restriction on dates of publication. Following 
screening of (a) titles and abstracts and (b) full 
papers, using pre-established inclusion and 
exclusion criteria, six papers were identified as 
meeting the criteria for inclusion (randomised 
controlled trial (RCT) or matched group 
comparison studies with baseline equivalence 
published in refereed journals). 

Data extraction and analyses were carried out 
on reading and language outcome measures 
comparing the Fast ForWord intervention 
groups to both active and untreated control 
groups. Results: M-eta-analyses indicated that 


For the 4 analyses of Fast ForWord compared to untreated 
control groups, the pooled effect size was .079 (95% Cl - 
.09 to .25), .17 (-.17 to .52) for passage comprehension, 
.01 (-.25 to .28) for receptive language and -.04 (95% -.33 
to .25) for expressive language. For comparisons with the 
treated control groups the equivalent pooled effect sizes 
were -.026 (95% Cl -.40 to .35), -.10 (-.40 to .21) for 
passage comprehension, .02 (-.27 to .31) for receptive 
language and -.06 (-.33 to .20) for expressive language. 
None of the 8 pooled effect sizes were reliably different 
from zero, and 4 of the effect sizes were actually negative 
(indicating worse performance in the Fast ForWord 
treatment group than the control group). Thus from the 
studies we have identified and analysed here there is no 
convincing evidence that Fast ForWord is effective in 
improving children’s single word reading, passage reading 
comprehension, receptive language or expressive 
language skills. 


39 


there was no significant effect of Fast ForWord 
on any outcome measure in comparison to 
active or untreated control groups. Conclusions: 
There is no evidence from the analysis carried 
out that Fast ForWord is effective as a 
treatment for children’s oral language or 
reading difficulties. 


Tokpah 

2008 

The Effects Of 
Computer Algebra 
Systems On 
Students’ 
Achievement In 
Mathematics 


0.38 


This meta-analysis sought to investigate the 
overall effectiveness of computer algebra 
systems (CAS) instruction, in comparison to 
non-CAS instruction, on students’ achievement 
in mathematics at pre-college and post- 
secondary institutions. The study utilized meta- 
analysis on 31 primary studies (102 effect 
sizes, N= 7,342) that were retrieved from online 
research databases and search engines, and 
explored the extent to which the overall 
effectiveness of CAS was moderated by various 
study characteristics. The overall effect size, 
0.38, was significantly different from zero. The 
mean effect size suggested that a typical 
student at the 50th percentile of a group taught 
using non-CAS instruction could experience an 
increase in performance to the 65th percentile, 
if that student was taught using CAS instruction. 
The fail-safe N, Nfs, hinted that 11,749 
additional studies with nonsignificant results 
would be needed to reverse the current finding. 
Three independent variables (design type, 
evaluation method, and time) were found to 
significantly moderate the effect of CAS. 

The current results do not predict future trends 
on the effectiveness of CAS; however, these 
findings suggest that CAS have the potential to 
improve learning in the classroom. Regardless 
of how CAS were used, the current study found 
that they contributed to a significant increase in 
students’ performance. 


The average effect size for CAS in a tutorial role (d = 0.40) 
did not differ significantly from the average effect size for 
CAS in a tool role (d=0.39), ns. 

The average effect sizes for studies that controlled for the 
effect of teacher (different teachers) and studies that did 
not control for the effect of teacher (same teacher) were 
found to be 0.41 and 0.30, respectively. 

The average effect size for studies in which CAS were 
used during evaluation (d = 0.31) was significantly lower 
than the average effect size for studies in which CAS were 
not used during evaluation (d = 0.42), QB(1) =4.35, p< 
0.05 

The average effect size for studies conducted from 1990 to 
1999 (d = 0.51) was significantly larger than the average 
effect size for studies conducted from 2000 to 2007 (d = 
0.24), x2(1) = 27.78, p < 0.05. 

While no other pair of comparison was significant, the 
difference between the average effect size for studies 
conducted in the 1980’s (d = 0.34) was less than that of 
studies conducted from 1990 to1999. 

Published studies (d = 0.38) unpublished studies (d = 
0.39) ns 


Torgerson & Zhu 
2003 


0.890 (C.I. 0.245 to 
1.535) — word 


What is the evidence for the effectiveness of 
ICT on literacy learning in English, 5-16? 


A range of five different kinds of ICT interventions emerged 
from the twelve included RCTs in the review: (1) computer- 
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A systematic review 
and meta-analysis of 
the effectiveness of 
ICT on literacy 
learning in English, 
5-16 


processing on 


writing 

0.204 (C.I. -0.168 to 
0.576) —ICT on 
spelling 


—0.047 (C.I. —0.33 to 
0.236) — Computer 
texts on reading 
comprehension/ 
questioning 

effect size 0.282 C.I. 
—0.003 to 0.566) 
Computer texts on 
reading 
comprehension/story 
retelling 


Studies were retrieved from the three electronic 
databases. Psyclnfo and ERIC were the richest 
sources for retrieving RCTs for this review. 
5.1.2 Mapping of all included studies Forty-two 
RCTs were identified for the effectiveness map. 
5.1.3 Nature of studies selected for 
effectiveness in-depth review The 12 included 
RCTs were assessed as being of ‘medium’ or 
‘high’ quality in terms of internal quality: ‘high’ 
quality in terms of relevance to the review; 
‘medium’ or ‘high’ in terms of the relevance of 
the topic focus; and ‘medium’ or high’ for 
overall weight of evidence. All 12 studies were 
undertaken in the USA with children aged 
between 5 and 14. Seven of the RCTs included 
samples where all or half of the participants 
experienced learning disabilities or difficulties or 
specific learning disabilities. All 12 studies 
focused on the psychological aspects or 
representations of literacy. 


assisted instruction (CAI), (2) networked computer system 
(classroom intranet), (3) word-processing software 
packages, (4) computer-mediated texts (electronic text) 
and (5) speech synthesis systems. There were also three 
literacy outcomes: (1) reading, including reading 
comprehension and phonological awareness (pre-reading 
understandings), (2) writing and (3) spelling. Six RCTs 
evaluated CAI interventions The CAI interventions 
consisted of studies designed to increase spelling abilities, 
reading abilities or phonological awareness (pre-reading 
understandings). One RCT evaluated a networked 
computer system intervention and two RCTs evaluated 
word-processing interventions; three RCTs evaluated 
computer mediated texts interventions and one RCT 
evaluated a speech synthesis intervention. In synthesis 
(1), for five different ICT interventions, overall we included 
20 comparisons from the 12 RCTs: 13 were positive and 
seven were negative. Of the positive ones, three were 
statistically significant, whilst of the seven negative trials, 
one was Statistically significant. These data would suggest 
that there is little evidence to support the widespread use 
of ICT in literacy learning in English. This also supports the 
findings from previous systematic reviews that have used 
data from rigorous study designs. It also supports the most 
recent observational data from the Impact2 study. These 
findings support the view that ICT use for literacy learning 
should be restricted to pupils participating in rigorous, 
randomised trials of such technology. In synthesis (2), we 
undertook three principal meta-analyses: one for each of 
the three literacy outcomes measures in which we were 
interested. In two, there was no evidence of benefit or 
harm; that is, in spelling and reading the small effect sizes 
were not statistically significant). In writing, there was weak 
evidence for a positive effect, but it was weak because 
only 42 children altogether were included in this meta- 
analysis. 


Torgerson and 
Elbourne 

2002 

A systematic review 


0.37 
Cl -0.02 to 0.77 


Recent Government policy in England and 
Wales on Information and Communication 
Technology (ICT) in schools is heavily 
influenced by a series of non-randomised 


No UK RCTs found. 

Our review found that the evidence base for the teaching 
of spelling by using a computer was very weak. It was 
particularly surprising that so few randomised controlled 
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and meta-analysis 

of the effectiveness of 
information and 
communication 
technology (ICT) on 
the teaching of 
spelling 


controlled studies. The evidence from these 
evaluations is equivocal with respect to the 
effect of ICT on literacy. In order to ascertain 
whether there is any effect of ICT on one small 
area of literacy, spelling, a systematic review of 
all randomised controlled trials (RCTs) was 
undertaken. Relevant electronic databases 
(including BEI, ERIC, Web of Science, 
PsycINFO, The Cochrane Library) were 
searched. Seven relevant RCTs were identified 
and included in the review. When six of the 
seven studies were pooled in a meta-analysis 
there was an effect, not statistically significant, 
in favour of computer interventions (Effect size 
= 0.37, 95% confidence interval =-0.02 to 0.77, 
p = 0.06). Sensitivity and sub-group analyses of 
the results did not materially alter findings. This 
review suggests that the teaching of spelling by 
using computer software may be as effective as 
conventional teaching of spelling, although the 
possibility of computer-taught spelling being 
inferior or superior cannot be confidently 
excluded due to the relatively small sample 
sizes of the identified studies. Ideally, large 
pragmatic randomised controlled trials need to 
be undertaken. 


trials had been undertaken in this area. This lack of 
evidence of effectiveness should not be interpreted as 
evidence that computer spelling programmes should 
instantly be withdrawn - the quality of the trials was 
variable, there may be unmeasured benefits, and there is 
no evidence that the programmes will harm children's 
spelling. Nevertheless, these conclusions are based on the 
best-available research appropriate to answering 
questions about the effectiveness of ICT on teaching and 
learning spelling. The onus should be on those wishing to 
introduce interventions such as these to first evaluate them 
formally, using rigorous research methods (large pragmatic 
RCTs), illuminated by the relevant theoretical 
developments. 


Vogel et al. 

2006 

Computer Gaming 
And Interactive 
Simulations For 
Learning: A Meta- 
Analysis 


0.07 


Substantial disagreement exists in the literature 
regarding which educational technology results 
in the highest cognitive gain for learners. In an 
attempt to resolve this dispute, we conducted a 
meta-analysis to decipher which teaching 
method, games and interactive simulations or 
traditional, truly dominates and under what 
circumstances. It was found that across people 
and situations, games and interactive 
simulations are more dominant for cognitive 
gain outcomes. However, consideration of 
specific moderator variables yielded a more 
complex picture. For example, males showed 
no preference while females showed a 
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preference for the game and interactive 
simulation programs. Also, when students 
navigated through the programs themselves, 
there was a significant preference for games 
and interactive simulations. However, when 
teachers controlled the programs, no significant 
advantage was found. Further, when the 
computer dictated the sequence of the 
program, results favored those in the traditional 
teaching method over the games and 
interactive simulations. These findings are 
discussed in terms of their implications for 
exiting theoretical positions as well as future 
empirical research. 


Waxman et al. 

2002 

A Quantitative 
Synthesis of Recent 
Research on the 
Effects of Teaching 
and Learning With 
Technology on 
Student Outcomes 


0.39 on cognitive 
outcomes 
Cl -.050 to .830 


To estimate the effects of teaching and learning 
with technology on students’ cognitive, 
affective, and behavioral outcomes of learning, 
138 effect sizes were calculated using statistical 
data from 20 studies that contained a combined 
sample of approximately 4,400 students. The 
mean of the study-weighted effect sizes 
averaging across all outcomes was .30 (p < 
.05), with a 95-percent confidence interval (Cl) 
of .004 - .598. This result indicates that 
teaching and learning with technology has a 
small, positive, significant (p < .05) effect on 
student outcomes when compared to traditional 
instruction. The mean study-weighted effect 
size for the 13 comparisons containing 
cognitive outcomes was .39, and the mean 
study-weighted effect size for the 60 
comparisons that focused on student affective 
outcomes was .208. On the other hand, the 
mean study-weighted effect size for the 30 
comparisons that contained behavioral 
outcomes was -.154, indicating that technology 
had a small, negative effect on students’ 
behavioral outcomes. The overall study- 
weighted effects were constant across the 
categories of study characteristics, quality of 


The relationship of each of the 56 conditioning (i-e., 
independent) variables to the mean study weighted effect 
size was tested for significance using ANOVA. The results 
indicate that none of the variables had a statistically 
significant (p < .01) impact on the study-weighted effect 
size. In other words, the overall findings suggest that the 
results do not differ significantly across categories of 
technology, instructional characteristics, methodological 
rigor, characteristics of the study, and subject 
characteristics. 
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study indicators, technology characteristics, and 
instructional/teaching characteristics. 


Waxman et al. 
2003 

A Meta-Analysis of 
the Effectiveness of 
Teaching and 
Learning With 
Technology on 
Student Outcomes 


(update of Waxman 
et al. 2002) 


0.44 (on cognitive 
outcomes) 
Cl .171 to .724 


To estimate the effects of teaching and learning 
with technology on students’ cognitive, 
affective, and behavioral outcomes of learning, 
282 effect sizes were calculated using statistical 
data from 42 studies that contained a combined 
sample of approximately 7,000 students. The 
mean of the study-weighted effect sizes 
averaging across all outcomes was .410 (p < 
.001), with a 95-percent confidence interval (Cl) 
of .175 to .644. This result indicates that 
teaching and learning with technology has a 
small, positive, significant (p < .001) effect on 
student outcomes when compared to traditional 
instruction. The mean study-weighted effect 
size for the 29 studies containing cognitive 
outcomes was .448, and the mean study- 
weighted effect size for the 10 comparisons that 
focused on student affective outcomes was 
.464. On the other hand, the mean study- 
weighted effect size for the 3 studies that 
contained behavioral outcomes was -.091, 
indicating that technology had a small, negative 
effect on students’ behavioral outcomes. The 
overall study-weighted effects were constant 
across the categories of study characteristics, 
quality of study indicators, technology 
characteristics, and instructional/teaching 
characteristics. 


The relationship of each of the 57 conditioning (i-e., 
independent) variables to the mean study-weighted effect 
size was tested for significance using ANOVA. The results 
indicate that none of the variables had a statistically 
significant (p < .01) impact on the study-weighted effect 
size. In other words, the overall findings suggest that the 
results do not differ significantly across categories of 
technology, instructional characteristics, methodological 
rigor, characteristics of the study, and subject 
characteristics. 
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Appendix 3: Summary table of Meta-analyses of the Impact of Computer and Digital Technologies on Attainment Published between 1990 and 1999 


Meta-analyses of the Impact of Computer and Digital Technologies on Attainment Published between 1990 and 1999 


Author/Title OverallES | Abstract Moderator variables 
Azevedo & Bernard | 0.80 A quantitative research synthesis (meta-analysis) was conducted on d (Rosenthal) (SD 0.57) 
1995 the literature concerning the effects of feedback on learning from 
A Meta-Analysis of computer-based instruction (CBI). Despite the widespread Feedback effects on learning and retention 
the Effects of acceptance of feedback in computerized instruction, empirical support | were found to vary with CBI typology, format 
Feedback in for particular types of feedback information has been inconsistent and | of unit content and access to supplemental 
Computer-Based contradictory. Effect size calculations from twenty-two studies materials. 
Instruction involving the administration of immediate achievement posttests 
resulted in a weighted mean effect size of .80. Also, a mean weighted 
effect size of .35 was obtained from nine studies involving delayed 
posttest administration (SD 0.17). Results indicate that the diagnostic 
and prescriptive management strategies of computer-based adaptive 
instructional systems provide the most effective feedback. The 
implementation of effective feedback in computerized instruction 
involves the computer's ability to verify the correctness of the learner's 
answer and the underlying causes of error. 
Bangert-Drowns 0.27 Word processing in writing instruction may provide lasting educational | (SE 0.11) 
1993 benefits to users because it encourages a fluid conceptualization of Frequency: once a week 0.04; 2-3 times per 
The Word Processor text and frees the writer from mechanical concerns. This meta- week 0.25; , more than 3 times a week 0.36. 
as an Instructional analysis reviews 32 studies that compared two groups of students Duration: 1-10 weeks -0.02; 11 to 20 weeks 
Tool: A Meta- receiving identical writing instruction but allowed only one group to 0.39; more than 20 weeks 0.28; 
Analysis of Word use word processing for writing assignments. Word processing Nine studies provided remedial writing 
Processing in groups, especially weaker writers, improved the quality of their writing. | instruction to students who had demonstrated 
Writing Instruction Word processing students wrote longer documents but did not have difficulty with writing. These nine studies 
more positive attitudes toward writing. More effective uses of word yielded an average effect size of 0.49. 
processing as an instructional tool might include adapting instruction Students using word processing during 
to software strengths and adding metacognitive prompts to the writing | writing instruction reliably begin to 
program. produce longer documents than students 
who do not have access to word processing. 
The average effect size for document length 
was 0.52 standard deviations. 
Becker NSPE Currently, schools are investing substantial funds in integrated Four randomized designs with ES 0.17 and 
1992 (No single learning systems (I.L.S.'s)—networked comprehensive basic skills ES 0.26. Other median ES for the different 
Computer-Based pooled software from a single vendor. Although rational arguments can be products varied were 0.0. 0.15, 0.17, 0.33, 
Integrated Learning | effect) made for the effectiveness of |.L.S.'s, districts want—and vendors are | 0.40. 


Systems In The 


supplying—empirical evidence for decision making. This article re- 
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Elementary and 
Middle Grades: A 
Critical Review and 
Synthesis of 
Evaluation Reports 


analyzes results reported in thirty evaluations of I.L.S.'s by using a 
common "effect size" statistic and correcting, where possible, for 
deficiencies in the original designs and reports. Some studies 
(including the most widely cited) substantially over-report |.L.S. 
effectiveness. On average, I.L.S.'s show a moderately positive effect 
on student achievement. However, the poor quality of most 
evaluations and the likely bias in what does get reported at all provide 
too weak a platform for district purchasing decisions. 


Fletcher-Flinn & 0.24 There has been a long-standing dispute about the efficacy of Glass’ A 
Gravatt computer assisted instruction (CAI) with regard to the interpretation of | Effect size did not differ significantly with 
1995 effect size estimates in reviews using techniques of meta-analysis. It educational level... 0.22 adults; 0.20 
The Efficacy of has been claimed that the data used to calculate these estimates secondary; 0.26 elementary; 0.55 Preschool 
Computer Assisted come from studies which are methodologically flawed. The aim of this | and K. 
Instruction (CAI): A study was to provide an updated meta-analysis on the learning effect | Special education 0.32 
Meta-Analysis of (CAI) over a broad range of study features with particular attention High ability 0.16; low ability 0.08 
focused on the effectiveness debate. Using standard procedures, the | Maths 0.32; Literacy 0.12; Science 0.26; Arts 
results and estimates were similar to previous reviews and showed a_ | 0.26 
learning benefit for CAl. The mean effect size for CAI was (.24) for the | Duration up to 4 weeks 0.22; more than 4 
years 1987-1992, with more recent studies showing an average of weeks 0.27 
(.33). Although moderate, these estimates tended to raise the Drill and practice 0.23; Simulation/ thinking 
average student from at least the 50th and 60th percentile. However, | 0.25; word processing 0.22 
studies which controlled for teacher and materials, and were of longer | Random assignment 0.23; non-random 0.25 
duration, and studies using pencil and paper equivalents of CAI 
showed no learning advantage over traditional forms of instruction. It 
is suggested that what accounts for the typical learning advantage of 
CAI in this meta-analysis and others is the better quality instruction 
provided by CAI materials. These materials seem versatile enough to 
be used effectively over a broad range of subjects and educational 
settings. While the materials did not seem to improve substantially 
over the past two decades as reflected by effect sizes, these 
estimates did not include the newer multimedia technology. It is 
concluded that educational approaches should be judged by a 
number of criteria including achievement gains and when this is done 
CAI may far surpass other forms of instruction. 
Khalili & Shashaani 0.38 A meta-analysis of 36 independent studies showed that computer Effect sizes ranged from -.88 to 1.54. The 
1994 (unweighted | applications hove a positive effect on students’ academic mean of the 151 effect sizes in this study was 
The effectiveness of | mean) achievement from elementary school to college. The average effect 38. 
computer size from 151 comparisons was .38; this indicates that use of Duration: 1-3 Weeks .14; 4-7 Weeks .94; 8- 


applications: A 
meta-analysis. 


computer applications raised students’ examination scores by .38 
standard deviation. Effects differed as a function of the computer 


11 Weeks .37; 12-15 Weeks .36; > 15 
Weeks .32 
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study feature. Effect sizes were higher in studies that used Logo 
programming language, when different teachers taught the 
experimental and the control group, when treatment was applied in a 
period of one to two months, and when subjects were selected from 
high schools. 


Type of use: CAI .37; Logo .45; Other 
Programming Languages .33; Drill and 
Practice .11; Tutorial .26; Simulation .79; 
Problem Solving .41; Unspecified .39 
Subject: Mathematics .52; Computer Science 
.28; Science .12; Reading/Language .17; 
Age: High schools was significantly larger than 
in all other groups: Elementary .34; Middle 
School .11; High School .62; College .45 
Supplement/ replacement: Replacement for 
Instruction .34; Supplement for Instruction .38 
Design: Pretest-Posttest .38; Posttest Only 
.38; Repeated Measurement 1 .45 

Random .29; Nonrandom .54 

Study Setting: Regular Classroom .27; 
Computer Lab .47 

Teacher: Same .35, Teacher .45 


Kuchler 0.28 The purpose of this meta-analysis is to integrate the findings 65 studies Min ES -0.47 to Max 2.59 Hedges g 

1998 (Hedges’ g) | contained in sixty-five primary studies, selected from published 0.32 Mean 0.44 SD 0.55 

The effectiveness of studies, ERIC documents, and dissertations, which investigated the Outliers removed: 61 studies Hedges’ g 0.28 

using computers to use of computers to teach secondary school mathematics in the Cl 0.17 to 0.39 (mean 0.36 0.42 SD Cl 0.25 to 

teach secondary United States during the last twenty years and to extract new 0.47) 

school (grades 6-12) knowledge from these studies. This analysis suggests that at the 0.55 for low attainers 

mathematics: A secondary school level computer-assisted instruction (CAI) has only Grade level not significant 

meta-analysis an overall small positive effect on mathematics achievement but a Low SES benefit most 
possible medium positive effect on retention of mathematical Random higher than non random allocation 
concepts and skills of secondary school students. LOGO as the most | From the perspective of implementation within 
effective. CAI is used the most frequently and is the most effective for | a classroom, CAI mathematics instruction 
teaching general mathematics courses. CAI appears to be equally appears to be the most effective when it is 
effective across gender and grade level, but has a greater positive used to supplement regular instruction 
impact on students from low socioeconomic backgrounds. From the (0.38), when the students work interactively 
perspective of research, CAl mathematics instruction appears to be on microcomputers located in the classroom, 
the most effective when subjects are randomly selected/assigned, when the students are homogeneously 
when commercial/standardized evaluation instruments are used, and _ | grouped by ability, when students work 
when the same instructor teaches all subjects. This analysis detects collaboratively (0.51 vs 0.21) in pairs, and 
no significant difference in mean CAI effectiveness between published | when the duration of the instruction is longer 
studies and dissertations and no significant trend in CAI effectiveness | than a semester (0.61 vs 0.38). 
over the period of this study. 

Kulik & Kulik 0.30 A meta-analysis of findings from 254 controlled evaluation studies Glass’ A SE 0.029 

1991 (unweighted | showed that computer-based instruction (CBI) usually produces Duration: Short duration (4 weeks or less) 
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Effectiveness of mean) positive effects on students. The studies covered learners of all age 0.42 (SE = 0.07); long 0.26 (SE = 0.03). 
Computer-Based levels -- from kindergarten pupils to adult students. CBI programs 
Instruction: An raised student examination scores by 0.30 standard deviations in the 
Updated Analysis average study, a moderate but significant effect. Size of effect varied, 
however, as a function of study feature. Effects were larger in 
published rather than unpublished studies, in studies in which 
different teachers taught experimental and control classes, and in 
studies of short duration. CBI also produced small but positive 
changes in student attitudes toward teaching and computers, and it 
reduced substantially the amount of time needed for instruction. 
Kulik 0.32 Introduction: What do evaluation studies say about computer-based Book chapter — analysis on the basis of 
1994 (unweighted | instruction? It is not easy to give a simple answer to the question. The | conceptual clarity of usage types (L1, L2 & 
Meta-analytic mean) term computer-based instruction has been applied to too many L3). L1 is general tech/no tech - 0.32; L2 — 
studies of findings different programs and the term evaluation ahs been used in too tutoring 0.38; L3 Stanford CCC 0.4 
on computer-based many different ways. Nonetheless, the question of what the research 97 studies 
instruction says cannot be ignored. Researchers want to know the answer, 0.32 equivalent to 3 months additional 
school administrators need to know and the public deserves to know. | progress 
How well has computer-based instruction worked? Conclusion: Meta- | Tutoring effective 
analysts have demonstrated repeatedly that programs of computer- Compares Computer tutoring with other 
based instruction usually have positive effects on student learning. approaches (p 24) - mid range. 
This conclusion has emerged from too many separate meta-analyses 
to be considered controversial. 
Lee 0.41 The purpose of this paper is to analyze evidence concerning the Glass’ A 
1999 (unweighted | effectiveness of simulation by examining the relationship between two | 7 of the 19 studies are elementary or high 
Effectiveness of mean) forms of simulations, pure and hybrid, and two modes of instructions, | school 
computer-based presentation and practice. A review of previous reviews is discussed 0.41 on academic achievement 
instructional concerning the effectiveness of instructional simulation. Via a meta- 
simulation: A meta analysis, 19 studies are examined. The meta-analysis leads to 
analysis following conclusions: 1. Within the presentation mode, the hybrid 
simulation is much more effective than the pure simulation. 2. 
Simulations are almost equally effective for both presentation and the 
practice modes if the hybrid simulation is used. 3. Specific guidance in 
simulation seems to help students to perform better. 4. When 
students learn in the presentation mode with the pure simulation, they 
showed a negative attitude toward simulation. 
Liao 0.48 A meta-analysis was performed to synthesize existing research Glass’ A 
1992 (unweighted | concerning the effects of computer-assisted instruction (CAI) on Range of study effects from -0.91 to 3.31. The 
Effects of computer- / mean) cognitive outcomes. Thirty-one studies were located from three overall grand mean for all 31 study-weighted 


assisted instruction 
on cognitive 


sources, and their quantitative data were transformed into Effect Size 
(ES). The analysis showed that 23 (74%) of 


ESs was 0.48. 
The overall grand median for all 31 study- 
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outcomes: A meta- 
analysis. 


the study-weighted ESs were positive and favored the CAI group over 
the control group. The overall grand mean of the study-weighted ESs 
for all 31 studies and 207 comparisons was 0.48; this suggests that 
students who had CAI experiences scored about 18 percentile points 
higher on various cognitive-ability tests than students who did not 
have CAI experiences. In 

addition, 6 of the 29 variables selected for this study had a statistically 
significant impact on the mean ES. The findings suggest that the 
outcomes of using CAI go beyond the content of that specific software 
or subject. 


weighted ESs was 0.34. The standard 
deviation of 0.91 reflects the great variability of 
ESs across studies. 

Duration: the mean comparison of studies in 
which the duration of treatment was less than 
6 months showed higher mean ESs than 
the mean ESs of studies in which duration 
of treatment was more than 6 months. 
Application type: tutorial software was 
significantly higher than drill-and-practice 
software, problem-solving software, and 
simulation (other differences non sig.) 


Liao 0.41 A meta-analysis was performed to synthesize existing research Glass’ A Study ES range -0.91 to 3.13. (SD 

1999 (unweighted | comparing the effects of hypermedia verse non-hypermedia 0.87) 

Effects of mean) instruction (e.g., CAI, text, traditional, videotape instruction) on Instrumentation - unspecified (researcher vs 

Hypermedia on students’ achievement. Forty-six studies were located from three standardised non sig.) 

Students’ sources, and their quantitative data were transformed into Effect Size | Type of research design: Single group 

Achievement: (ES). The overall grand mean of the study-weighted ES for all 46 repeated measures sig. higher (pretest- 

A Meta-Analysis studies was 0.41. The results suggest that hypermedia instruction is posttest control group, nonequivalent control 
more effective when there is no instruction for the comparison group group, and posttest only control group designs 
or when the comparison group used videotape instruction. However, all non sig.) 

CAI and text instructions are slightly more effective than hypermedia Type of delivery system: simulators 
instruction. As a whole, the results of this analysis suggest that the significantly higher than interactive 
effects of hypermedia instruction on students' achievement are mixed, | multimedia 
depends on what type of instruction it compares to. In addition, four of | Age: non sig. 
the seventeen variables selected for this study (i.e., instrumentation, Supplement: ES for supplement group was 
type of research design, type of delivery system, and comparison 0.18 SD higher than the replacement group 
group) had a statistically significant impact on the mean ES. Duration: The mean ESs for studies lasting 
1-4 
months or less than 1 week were higher, 
while the mean ESs for studies lasting 1-4 
weeks or over 4 months were lower. 

Ryan 0.31 This meta-analysis was undertaken to clarify existing knowledge of Glass’ A 

1991 (unweighted | computer use in instruction and to provide information concerning 40 studies 58 effects 

Meta-Analysis of mean) implementation factors that would be helpful to educational Mean 0.309 Median 0.296 range -0.482 to 


Achievement Effects 
of Microcomputer 
Applications in 
Elementary Schools 


administrators. A meta-analysis technique was used to synthesize the 
results of 40 independent studies. Variables analyzed included 
characteristics of students, teachers, physical settings, and 
instructional formats. 


1.226 

Effect sizes higher with more than 10 hours 
training or CPD (0.40) 

Teacher written software 0.82 higher than 
commercial 0.29 


49 


Appendix 4: Bibliography of Meta-analyses of the Impact of Computer and Digital 
Technologies on Education 


Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in 
computer-based instruction. Journal of Educational Computing Research, 13.2: 111- 
127. 

Bangert-Drowns, R. L. (1993). The word processor as an instructional tool: A meta-analysis of 
word processing in writing instruction. Review of Educational Research, 63.1: 69-93. 

Bangert-Drowns, R. L., Kulik, J. A., & Kulik, C. C. (1985). Effectiveness of computer-based 
education in secondary schools. Journal of Computer-Based Instruction, 12.3: 59-68. 

Bayraktar, S. (2000). A meta-analysis on the effectiveness of computer-assisted instruction in 
science education. Journal of Research on Technology in Education, 34.2: 173-189. 

Becker, H. J. (1992). Computer-based integrated learning systems in the elementary and 
middle grades: A Critical Review and Synthesis of Evaluation Reports Journal of 
Educational Computing Research, 8.1: 1-41. 

Blok, H., Oostdam, R., Otter, M. E., & Overmaat, M. (2002). Computer-assisted instruction in 
support of beginning reading instruction: A review. Review of Educational Research, 
72.1: 101-130. 

Camnalbur, M., & Erdogan, Y. (2008). A Meta Analysis on the Effectiveness of Computer- 
Assisted Instruction: Turkey Sample. Educational Sciences: Theory and Practice, 8.2: 
497-505. 

Cassil, K.M. (2005) A Meta Analysis: The Effectiveness Of The Use Of Mobile Computers On 
The Attitude And Academic Outcomes Of K-12 Students (Doctoral dissertation, 
Auburn, Alabama) ProQuest Dissertations and Theses. 

Cheung, A. C., & Slavin, R. E. (2011). The effectiveness of educational technology 
applications for enhancing mathematics achievement in K-12 classrooms: A meta- 
analysis. Johns Hopkins University: Best Evidence Encyclopedia, Center for Data- 
Driven Reform in Education. 

Christmann, E. P. & Badgett, J. L. (2003). A meta-analytic comparison of the effects of 
computer-assisted instruction on elementary students’ academic achievement. 
Information Technology in Childhood Education Annual, 2003(1), 91-104. 

Fletcher-Flinn, C.M. And Gravatt, B. (1995). The Efficacy Of Computer Assisted Instruction 
(CAI): A Meta-Analysis Journal Of Educational Computing Research 12.3: 219-242. 

Goldberg, A., Russell, M. & Cook, A. (2003). The Effect of Computers on Student Writing:- A 
Meta-Analysis of .Studies from 1992 to 2002 The Journal of Technology, Learning, and 
Assessment 2. 1: 2- 51. 

Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. 
Journal of Educational Psychology, 99.3: 445-476. 

Khalili, A., & Shashaani, L. (1994). The Effectiveness Of Computer Applications : A Meta- 
Analysis. Journal of Research on Computing in Education, 27.1: 48-62. 

Kuchler, J. M. (1998). The effectiveness of using computers to teach secondary school 
(grades 6-12) mathematics: a meta-analysis (Doctoral dissertation, University of 
Massachusetts, Lowell) ProQuest Dissertations and Theses. 

Kulik, J. (1994). Meta-analytic studies of findings on computer-based instruction. In Baker, E. 
L. & O'Neil, H. F. Ur. (Eds.), Technology assessment in education and training. (pp. 9- 
33) Hillsdale, NJ: Lawrence Erlbaum. 

Kulik, C. C., & Kulik, J. A. (1991). Effectiveness of computer-based education in elementary 
schools. Computers in Human Behavior, 7: 75-94. 

Kulik, J. A., Kulik, C. C., & Bangert-Drowns, R. L. (1985). Effectiveness of computer-based 
education in elementary schools. Computers in Human Behavior, 1.1: 59-74. 

Kulik, J.A. (2003). Effects of Using Instructional Technology in Elementary and Secondary 
Schools : What Controlled Evaluation Studies Say. SRI Project: Final Report. Arlington, 
Va. SRI International (82p) 

Lee, J. (1999). Effectiveness of computer-based instructional simulation: a meta-analysis. 
International Journal of Instructional Media, 26.1: 71-85. 

LeJeune, J. V. (2002). A meta-analysis of outcomes from the use of computer-simulated 
experiments in science education. (Doctoral dissertation, Texas A&M University) 
ProQuest Dissertations and Theses. 

Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school 
students’ mathematics learning. Educational Psychology Review, 22.3: 215-243. 


50 


Liao, Y. (1992). Effects of computer-assisted instruction on cognitive outcomes: A meta- 
analysis. Journal of Research on Computing in Education, 24.3: 367-380. 

Liao, Y. (1999). Effects of hypermedia on students’ achievement: A meta-analysis. Journal of 
Educational Multimedia and Hypermedia, 8.3: 255-277. 

Liao, Y. (2005). Effects of computer-assisted instruction on students’ achievement in Taiwan: 
A meta-analysis Computers & Education 48: 216-233. 

Lou, Y., Abrami, P. C., & d’ Apollonia, S. (2001). Small Group and Individual Learning with 
Technology: A Meta-Analysis. Review of Educational Research, 71.3: 449-521. 

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence- 
based practices in online learning: A meta-analysis and review of online learning 
studies. Washington DC: U.S. Department of Education Office of Planning, Evaluation, 
and Policy Development Policy and Program Studies Service 

Moran, J., Ferdig, R. E., Pearson, P. D., Wardrop, J., & Blomeyer, R. L. (2008). Technology 
and reading performance in the middle-school grades: A meta-analysis with 
recommendations for policy and practice. Journal of Literacy Research, 40.1: 6-58. 

Morphy, P., & Graham, S. (2012). Word processing programs and weaker writers/readers: a 
meta-analysis of research findings. Reading and Writing, 25.3: 641-678. 

Niemiec, R. P., & Walberg, H. J. (1985). Computers and Achievement in the Elementary 
Schools. Journal of Educational Computing Research, 1.4: 435- 440. 

Onuoha, C. O. (2007). Meta-analysis of the effectiveness of computer-based laboratory 
versus traditional hands-on laboratory in college and pre-college science instructions. 
(Doctoral Dissertation Capella University, Minneapolis, MN.) ProQuest Dissertations 
and Theses. 

Pearson, D. P., Ferdig, R. E., Blomeyer, R. L., & Moran, J. (2005). The effects of technology 
on reading performance in the middle-school grades: A meta-analysis with 
recommendations for policy. Napierville, Il.: Learning Point Associates. 

Rosen, Y. & Salomon, G. (2007). The Differential Learning Achievements Of Constructivist 
Technology Intensive Learning Environments As Compared With Traditional Ones: A 
Meta-Analysis Journal Of Educational Computing Research, 36.1: 1-14. 

Ryan, A. W. (1991). Meta-analysis of achievement effects of microcomputer applications in 
elementary schools. Educational Administration Quarterly, 27.2: 161-184. 

Sandy-Hanson, A. E. (2006). A meta-analysis of the impact of computer technology versus 
traditional instruction on students (Doctoral dissertation, Howard University, 
Washington DC.) ProQuest Dissertations and Theses. 

Seo, Y.-J., & Bryant, D. P. (2009). Analysis of studies of the effects of computer-assisted 
instruction on the mathematics performance of students with learning disabilities. 
Computers & Education, 53.3: 913-928. 

Sisson, C. B. (2008). A Meta-analytic Investigation Into the Efficacy of Fast ForWord 
Intervention on Improving Academic Performance. (Doctoral dissertation Regent 
University, Virginia Beach, VA.) ProQuest Dissertations and Theses. 

Sitzmann, T., Kraiger, K., Stewart, D. & Wisher, R. (2006). The Comparative Effectiveness Of 
Web-Based And Classroom Instruction: A Meta-Analysis Personnel Psychology 59: 
623-664. 

Soe, K., Koki, S., & Chang, J. M. (2000). Effect of computer-assisted instruction (CAI) on 
reading achievement: A meta-analysis. Pacific Resources for Education and Learning, 
Honolulu, HI. Retrieved from: http://www.prel.org/products/products/Effect-CAI.pdf 

Strong, G. K., Torgerson, C. J., Torgerson, D., & Hulme, C. (2011). A systematic meta- 
analytic review of evidence for the effectiveness of the ‘Fast ForWord’language 
intervention program. Journal of Child Psychology and Psychiatry, 52.3: 224-235. 

Tamim, R. M. (2009). Effects of Technology on Students’ Achievement: A Second-Order 
Meta-Analysis. (Doctoral dissertation, Concordia University). Retrieved from 
http:/Awww.emeraldinsight.com/10.1108/eb047931 

Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What 
Forty Years of Research Says About the Impact of Technology on Learning A Second- 
Order Meta-Analysis and Validation Study. Review of Educational Research, 81.1: 4- 
28. 

Tokpah, C. L. (2008). The Effects Of Computer Algebra Systems On Students’achievement In 
Mathematics (Doctoral dissertation, Kent State University). ProQuest Dissertations and 
Theses. 


51 


Torgerson C.J. & Elbourne D. (2002) A Systematic Review and Meta-Analysis of the 
Effectiveness of Information and Communication Technology (ICT) on the Teaching of 
Spelling. Journal of Research in Reading 25 :129-43. 

Torgerson, C., & Zhu, D. (2003). A systematic review and meta-analysis of the effectiveness 
of ICT on literacy learning in English, 5-16. EPPl-Centre, Social Science Research 
Unit, Institute of Education, University of London. 

Vogel, J.J., Vogel, D.S., Cannon-Bowers, J., Bowers, C.A, Muse, K. and Wright, M. (2006). 
Computer Gaming And Interactive Simulations For Learning: A Meta-Analysis Journal 
of Educational Computing Research, 34.3: 229-243. 

Waxman, H. C., Connell, M. L., & Gray, J. (2002). A Quantitative Synthesis of Recent 
Research on the Effects of Teaching and Learning With Technology on Student 
Outcomes. (28p). Naperville, Illinois: North Central Regional Laboratory 

Waxman, H. C., Lin, Meng.-Fen, & Michko, G. M. (2003). A Meta-Analysis of the 
Effectiveness of Teaching and Learning With Technology on Student Outcomes 
December 2003. Naperville, Illinois: Learning Point Associates. 


52 


